Nov 02 14:54:49.455288 ip-10-0-135-181 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 14:54:49.455301 ip-10-0-135-181 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 14:54:49.455310 ip-10-0-135-181 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 14:54:49.455637 ip-10-0-135-181 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 14:55:00.887923 ip-10-0-135-181 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 14:55:00.887943 ip-10-0-135-181 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot df614fbc08c748f2917a47bd3bd47427 -- Nov 02 14:57:03.513271 ip-10-0-135-181 systemd[1]: Starting Kubernetes Kubelet... Nov 02 14:57:03.955178 ip-10-0-135-181 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:57:03.955178 ip-10-0-135-181 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 14:57:03.955178 ip-10-0-135-181 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:57:03.955178 ip-10-0-135-181 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:57:03.959014 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.958926 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:03.959014 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959014 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959019 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959022 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959026 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959029 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959031 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959034 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959037 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959041 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959043 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959046 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959049 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959051 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959054 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959057 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959060 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959062 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959065 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959068 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959076 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:03.959086 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959080 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959083 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959086 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959088 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959091 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959094 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959097 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959100 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959103 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959106 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959108 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959111 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959114 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959116 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959119 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959122 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959124 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959127 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959130 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959132 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:03.959561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959135 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959137 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959140 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959142 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959146 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959148 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959151 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959154 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959156 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959159 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959161 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959164 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959166 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959169 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959172 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959175 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959177 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959180 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959183 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:03.960054 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959186 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959188 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959191 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959193 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959197 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959202 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959205 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959208 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959211 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959213 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959216 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959218 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959221 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959224 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959226 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959229 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959231 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959234 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959236 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959239 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:03.960523 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959241 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959244 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959247 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959249 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959252 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959254 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959259 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959263 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959266 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959268 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959271 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959275 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959279 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959282 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959285 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959288 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959291 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959293 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959297 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:03.961124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959299 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959302 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959304 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959307 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959310 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959312 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.959315 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960093 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960101 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960104 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960107 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960110 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960113 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960116 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960119 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960121 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960124 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960127 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960129 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:03.961799 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960132 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960136 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960139 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960142 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960145 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960148 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960150 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960153 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960156 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960159 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960161 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960164 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960166 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960169 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960172 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960174 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960177 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960179 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960182 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:03.962305 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960185 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960187 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960190 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960192 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960195 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960198 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960201 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960204 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960206 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960209 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960211 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960214 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960216 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960219 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960222 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960225 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960228 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960231 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960233 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:03.962757 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960236 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960238 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960241 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960243 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960246 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960248 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960251 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960255 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960259 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960262 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960265 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960267 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960270 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960273 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960275 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960278 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960280 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960283 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960286 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:03.963229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960288 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960291 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960294 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960296 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960299 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960301 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960304 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960307 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960309 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960312 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960316 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960319 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960322 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960324 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960327 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960330 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960332 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960335 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960337 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960339 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:03.963671 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960342 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960344 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960347 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960349 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960352 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960354 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960357 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960359 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960364 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960368 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960372 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960375 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960379 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960382 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960385 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960388 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.960391 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.961805 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.961857 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:03.964156 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.961889 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.961906 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.961910 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962046 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962051 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962056 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962062 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962067 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962071 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962073 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962076 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962080 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962083 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962086 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962089 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962091 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962094 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962096 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962099 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962102 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:03.964606 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962104 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962107 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962110 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962112 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962117 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962120 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962122 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962125 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962128 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962130 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962133 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962136 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962139 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962141 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962144 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962148 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962150 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962153 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962156 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:03.965124 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962160 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962165 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962168 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962171 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962174 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962177 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962180 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962183 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962188 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962190 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962193 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962196 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962199 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962201 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962205 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962209 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962212 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962215 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962219 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962221 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:03.965567 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962225 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962227 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962230 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962233 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962236 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962239 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962241 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962244 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962246 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962249 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962252 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962255 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962257 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962260 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962262 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962265 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962268 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962270 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962273 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962275 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:03.966152 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962278 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962280 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962283 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962286 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962288 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962291 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962293 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962296 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962299 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962302 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962306 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962309 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962312 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962315 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962317 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962320 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962323 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962326 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962329 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962331 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:03.966656 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962334 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962336 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962339 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962342 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.962345 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963586 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963596 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963604 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963609 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963614 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963618 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963623 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963628 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963631 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963635 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963638 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963642 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963645 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963648 2577 flags.go:64] FLAG: --cgroup-root="" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963651 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963654 2577 flags.go:64] FLAG: --client-ca-file="" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963657 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963660 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 14:57:03.967136 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963666 2577 flags.go:64] FLAG: --cluster-domain="" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963669 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963672 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963676 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963679 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963683 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963686 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963689 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963693 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963696 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963699 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963702 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963705 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963708 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963712 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963715 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963718 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963721 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963724 2577 flags.go:64] FLAG: --enable-server="true" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963727 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963756 2577 flags.go:64] FLAG: --event-burst="100" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963760 2577 flags.go:64] FLAG: --event-qps="50" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963763 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963767 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 14:57:03.967710 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963770 2577 flags.go:64] FLAG: --eviction-hard="" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963775 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963778 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963781 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963784 2577 flags.go:64] FLAG: --eviction-soft="" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963787 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963790 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963793 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963796 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963799 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963802 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963805 2577 flags.go:64] FLAG: --feature-gates="" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963809 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963812 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963815 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963819 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963822 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963825 2577 flags.go:64] FLAG: --help="false" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963828 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963831 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963834 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963837 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963840 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 14:57:03.968320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963843 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963846 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963849 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963852 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963855 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963858 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963861 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963864 2577 flags.go:64] FLAG: --kube-reserved="" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963867 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963870 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963873 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963877 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963880 2577 flags.go:64] FLAG: --lock-file="" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963882 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963885 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963888 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963894 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963897 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963900 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963903 2577 flags.go:64] FLAG: --logging-format="text" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963906 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963909 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963912 2577 flags.go:64] FLAG: --manifest-url="" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963915 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963920 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 14:57:03.968861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963923 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963927 2577 flags.go:64] FLAG: --max-pods="110" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963930 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963933 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963936 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963939 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963942 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963945 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963948 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963956 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963959 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963962 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963965 2577 flags.go:64] FLAG: --pod-cidr="" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963985 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963989 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963992 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963995 2577 flags.go:64] FLAG: --port="10250" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.963998 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964001 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ac3bb78af063003e" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964008 2577 flags.go:64] FLAG: --qos-reserved="" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964011 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964014 2577 flags.go:64] FLAG: --register-node="true" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964017 2577 flags.go:64] FLAG: --register-with-taints="" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964021 2577 flags.go:64] FLAG: --registry-burst="10" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964024 2577 flags.go:64] FLAG: --registry-qps="5" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964027 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 02 14:57:03.969454 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964030 2577 flags.go:64] FLAG: --reserved-memory="" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964034 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964037 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964040 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964043 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964045 2577 flags.go:64] FLAG: --runonce="false" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964048 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964051 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964054 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964057 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964060 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964063 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964066 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964070 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964073 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964075 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964078 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964081 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964084 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964091 2577 flags.go:64] FLAG: --system-cgroups="" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964095 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964101 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964104 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964107 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964112 2577 flags.go:64] FLAG: --tls-min-version="" Nov 02 14:57:03.970128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964115 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964119 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964122 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964125 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964128 2577 flags.go:64] FLAG: --v="2" Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964133 2577 flags.go:64] FLAG: --version="false" Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964138 2577 flags.go:64] FLAG: --vmodule="" Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964142 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964145 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964284 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964288 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964292 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964295 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964298 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964301 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964305 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964309 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964312 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964315 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964317 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964320 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:03.970701 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964323 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964325 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964328 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964331 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964334 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964339 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964341 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964344 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964347 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964349 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964352 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964355 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964357 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964360 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964363 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964366 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964368 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964371 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964373 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964376 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964378 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:03.971229 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964381 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964383 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964386 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964389 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964391 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964394 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964396 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964399 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964401 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964404 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964407 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964409 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964412 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964414 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964417 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964420 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964424 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964427 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964430 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:03.971721 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964433 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964435 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964438 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964442 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964445 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964448 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964451 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964455 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964458 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964460 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964463 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964467 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964469 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964472 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964475 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964477 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964480 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964482 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964485 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:03.972208 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964487 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964490 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964492 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964495 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964498 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964500 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964502 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964505 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964508 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964510 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964515 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964518 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964521 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964524 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964527 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964529 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964532 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964535 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964537 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964540 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:03.972646 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964547 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964549 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964552 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964555 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964557 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964560 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964562 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964565 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964567 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964570 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964572 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964575 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964578 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964580 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964583 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:03.973197 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.964610 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964647 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964651 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964654 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964657 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964660 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964662 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964667 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964671 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964673 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964676 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964679 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964682 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964684 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964687 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964689 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964692 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964695 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964698 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:03.973546 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964701 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964704 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964706 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964710 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964714 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964718 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964720 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964723 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964725 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964728 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964731 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964734 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964737 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964739 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964742 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964744 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964747 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964749 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964752 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:03.974023 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964754 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964758 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964762 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964765 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964767 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964770 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964772 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964775 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964777 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964780 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964783 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964785 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964788 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964791 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964794 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964797 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964799 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964802 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964805 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964807 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:03.974453 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964810 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964813 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964815 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964818 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964821 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964823 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964826 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964828 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964831 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964833 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964836 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964839 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964841 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964845 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964849 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964851 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964854 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964857 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964860 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:03.974916 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964862 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964865 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964869 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964872 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964875 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964878 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964881 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964884 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964887 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964889 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964892 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964894 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964897 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964899 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964902 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964905 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964907 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964909 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964912 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964915 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:03.975372 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964917 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964920 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964922 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964925 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964927 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964930 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964934 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964937 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964940 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.964942 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.972074 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.972089 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.972906 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.972911 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.972969 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.972988 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.972992 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.972995 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.972998 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973002 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:03.975831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973005 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973008 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973011 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973013 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973016 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973020 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973024 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973028 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973031 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973034 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973037 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973039 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973042 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973044 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973047 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973050 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973052 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973055 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973057 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:03.976360 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973060 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973062 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973065 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973068 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973072 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973075 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973078 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973081 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973083 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973086 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973089 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973091 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973094 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973097 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973099 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973102 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973104 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973107 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973110 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:03.976793 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973112 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973115 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973118 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973120 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973123 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973125 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973128 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973131 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973133 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973136 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973138 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973141 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973144 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973146 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973149 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973152 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973154 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973159 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973162 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973165 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:03.977313 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973167 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973170 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973172 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973175 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973178 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973180 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973183 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973185 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973189 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973192 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973195 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973198 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973201 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973204 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973206 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973209 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973212 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973214 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973217 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973220 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:03.977773 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973222 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973225 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973227 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973230 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973233 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973235 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973238 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973241 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973243 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973246 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973249 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973252 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973255 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973257 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973260 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973262 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973265 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973268 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973271 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973273 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:03.978257 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973276 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973278 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.973306 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973460 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973465 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973469 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973471 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973474 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973476 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973479 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973481 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973484 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973487 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973490 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973492 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973494 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:03.978716 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973497 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973500 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973503 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973505 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973508 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973511 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973514 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973516 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973519 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973522 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973525 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973527 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973530 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973532 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973535 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973537 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973540 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973542 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973545 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973547 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:03.979101 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973550 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973553 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973556 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973559 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973561 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973564 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973567 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973569 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973572 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973574 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973577 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973580 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973582 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973586 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973590 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973592 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973596 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973598 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973601 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:03.979561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973603 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973606 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973609 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973612 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973614 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973617 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973619 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973622 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973624 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973627 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973629 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973632 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973635 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973637 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973640 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973642 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973645 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973647 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973650 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973652 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:03.980055 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973655 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973657 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973660 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973663 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973665 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973668 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973670 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973673 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973675 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973678 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973680 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973683 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973686 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973688 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973691 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973694 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973696 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973699 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973702 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973704 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:03.980512 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973707 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973709 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973712 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973714 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973717 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973719 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973722 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973725 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973727 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973731 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973734 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973736 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973739 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973742 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.973764 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:57:03.980959 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973799 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973802 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973805 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973808 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973811 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973814 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973816 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973820 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973824 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973827 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973831 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973833 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973836 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973839 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973842 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973845 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973847 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973850 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973852 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:03.981381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973855 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973858 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973861 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973863 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973866 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973870 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973873 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973876 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973878 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973881 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973884 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973886 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973889 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973891 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973894 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973897 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973899 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973902 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973905 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973907 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:03.981810 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973910 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973912 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973915 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973918 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973921 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973923 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973926 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973929 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973932 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973934 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973937 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973939 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973942 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973944 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973947 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973949 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973952 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973956 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973959 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:03.982292 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973963 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973966 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973969 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973987 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973991 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973994 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.973997 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974000 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974002 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974005 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974008 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974016 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974019 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974022 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974025 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974028 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974030 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974033 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974036 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:03.982727 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974038 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974042 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974045 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974048 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974050 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974053 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974056 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974058 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974061 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974064 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974067 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974072 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974074 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974077 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974079 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974082 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974084 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974088 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974090 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974095 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:03.983198 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974097 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974100 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974102 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974105 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974108 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974110 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974113 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974115 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:03.974118 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.974174 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.976337 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.977661 2577 server.go:1014] "Starting client certificate rotation" Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.977757 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.978601 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.980219 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:57:03.983647 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.981287 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:57:03.994876 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.994858 2577 log.go:25] "Validated CRI v1 runtime API" Nov 02 14:57:03.998384 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.998367 2577 log.go:25] "Validated CRI v1 image API" Nov 02 14:57:03.999320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:03.999296 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 14:57:04.001905 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.001887 2577 fs.go:135] Filesystem UUIDs: map[022ea997-829b-4a43-ba1b-99028c13e25c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 80488794-8acc-48f4-82ba-39a0f30dedd5:/dev/nvme0n1p4] Nov 02 14:57:04.001998 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.001904 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 14:57:04.008016 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.007899 2577 manager.go:217] Machine: {Timestamp:2026-11-02 14:57:04.007433112 +0000 UTC m=+0.369259073 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099249 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c34a9bf8814896a57e21359c25744 SystemUUID:ec2c34a9-bf88-1489-6a57-e21359c25744 BootID:df614fbc-08c7-48f2-917a-47bd3bd47427 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e8:76:c9:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e8:76:c9:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:f1:8e:b0:61:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 14:57:04.008016 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.008012 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 14:57:04.008123 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.008078 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 14:57:04.011551 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.011526 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 14:57:04.011685 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.011554 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-181.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 14:57:04.011725 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.011687 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 14:57:04.011725 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.011696 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 14:57:04.011725 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.011715 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:57:04.011795 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.011730 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:57:04.012658 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.012648 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 14:57:04.016190 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.016178 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 14:57:04.016308 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.016299 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 14:57:04.017095 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.017086 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 02 14:57:04.017123 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.017098 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 14:57:04.017123 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.017109 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 14:57:04.017123 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.017117 2577 kubelet.go:405] "Adding apiserver pod source" Nov 02 14:57:04.017216 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.017125 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 14:57:04.017465 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.017442 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:57:04.020025 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.020008 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 14:57:04.025569 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.025557 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:57:04.026431 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.026419 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 14:57:04.026476 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.026448 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 14:57:04.026598 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.026587 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 14:57:04.026631 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.026602 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 14:57:04.026631 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.026609 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 14:57:04.026631 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.026614 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 14:57:04.026631 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.026620 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 14:57:04.026631 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.026625 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 14:57:04.027209 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.027201 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 14:57:04.027245 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.027210 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 14:57:04.027245 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.027217 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 14:57:04.027245 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.027223 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 14:57:04.027245 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.027237 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 14:57:04.027356 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.027247 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 14:57:04.028150 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.028140 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 14:57:04.028150 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.028149 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 14:57:04.031773 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.031761 2577 server.go:1263] "Started kubelet" Nov 02 14:57:04.031851 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.031827 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 14:57:04.031905 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.031867 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 14:57:04.031954 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.031922 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 14:57:04.032413 ip-10-0-135-181 systemd[1]: Started Kubernetes Kubelet. Nov 02 14:57:04.033625 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.033605 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 14:57:04.034985 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.034708 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 02 14:57:04.040235 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.039814 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 14:57:04.043904 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.043886 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 14:57:04.044546 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.044521 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 14:57:04.044775 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.044693 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.045421 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.045402 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 14:57:04.045421 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.045418 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 14:57:04.045564 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.045405 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 14:57:04.045564 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.045524 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 02 14:57:04.045564 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.045532 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 02 14:57:04.046339 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.046321 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 14:57:04.046339 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.046336 2577 factory.go:55] Registering systemd factory Nov 02 14:57:04.046339 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.046343 2577 factory.go:223] Registration of the systemd container factory successfully Nov 02 14:57:04.046522 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.046510 2577 factory.go:153] Registering CRI-O factory Nov 02 14:57:04.046567 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.046524 2577 factory.go:223] Registration of the crio container factory successfully Nov 02 14:57:04.046567 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.046537 2577 factory.go:103] Registering Raw factory Nov 02 14:57:04.046567 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.046546 2577 manager.go:1199] Started watching for new ooms in manager Nov 02 14:57:04.047636 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.047624 2577 manager.go:319] Starting recovery of all containers Nov 02 14:57:04.059378 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.059363 2577 manager.go:324] Recovery completed Nov 02 14:57:04.065019 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.065002 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:04.066776 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.066761 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:04.066874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.066793 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:04.066874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.066806 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:04.067322 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.067306 2577 cpu_manager.go:226] "Starting" policy="none" Nov 02 14:57:04.067322 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.067318 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:04.067415 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.067321 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 14:57:04.067415 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.067342 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 14:57:04.068438 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.068420 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:04.068503 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.068456 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:04.068503 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.068473 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:04.068832 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.068819 2577 policy_none.go:50] "Start" Nov 02 14:57:04.068873 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.068836 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 14:57:04.068873 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.068849 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 14:57:04.070083 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.070070 2577 policy_none.go:44] "Start" Nov 02 14:57:04.148228 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.144803 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.168734 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.168709 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-181.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 14:57:04.169951 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.168702 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-181.us-west-2.compute.internal.18e44246ef63c1f3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-181.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-181.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:57:04.031736307 +0000 UTC m=+0.393562265,LastTimestamp:2026-11-02 14:57:04.031736307 +0000 UTC m=+0.393562265,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-181.us-west-2.compute.internal,}" Nov 02 14:57:04.177360 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.177322 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 14:57:04.178431 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.178416 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 14:57:04.179079 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.179061 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 14:57:04.179182 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.179120 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 14:57:04.179182 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.179158 2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 14:57:04.241200 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.241160 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-181.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 14:57:04.241334 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.241208 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-181.us-west-2.compute.internal.18e44246f17a70c5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-181.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-181.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-181.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:57:04.066777285 +0000 UTC m=+0.428603246,LastTimestamp:2026-11-02 14:57:04.066777285 +0000 UTC m=+0.428603246,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-181.us-west-2.compute.internal,}" Nov 02 14:57:04.241438 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.241416 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-181.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 14:57:04.243587 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.243559 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 14:57:04.244892 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.244875 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.245947 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.245935 2577 manager.go:341] "Starting Device Plugin manager" Nov 02 14:57:04.246042 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.246028 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 14:57:04.246083 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.246043 2577 server.go:87] "Starting device plugin registration server" Nov 02 14:57:04.246253 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.246243 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 14:57:04.246289 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.246254 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 14:57:04.246388 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.246370 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 14:57:04.246478 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.246448 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 14:57:04.246478 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.246456 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 14:57:04.246997 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.246962 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 14:57:04.247065 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.247020 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.266916 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.266901 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 14:57:04.266987 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.266952 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zkv8b" Nov 02 14:57:04.280056 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.280036 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal"] Nov 02 14:57:04.280127 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.280087 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:04.280862 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.280846 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:04.280924 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.280874 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:04.280924 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.280884 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:04.282287 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.282276 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:04.282419 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.282405 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.282456 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.282432 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:04.282936 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.282920 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:04.283017 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.282952 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:04.283017 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.282963 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:04.283017 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.282920 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:04.283017 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.283010 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:04.283127 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.283024 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:04.284373 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.284354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.284373 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.284377 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:04.285040 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.285025 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:04.285125 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.285055 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:04.285125 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.285070 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:04.308791 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.308770 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" node="ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.314318 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.314302 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" node="ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.319924 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.319906 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 14:57:04.319924 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.319916 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 02 14:57:04.320048 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.319941 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-181.us-west-2.compute.internal.18e44246f17ac836 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-181.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-181.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-181.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:57:04.06679967 +0000 UTC m=+0.428625627,LastTimestamp:2026-11-02 14:57:04.06679967 +0000 UTC m=+0.428625627,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-181.us-west-2.compute.internal,}" Nov 02 14:57:04.346746 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.346724 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:04.346884 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.346862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/010a8efd81cfe8476c37b8d73ee870b2-config\") pod \"kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"010a8efd81cfe8476c37b8d73ee870b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.346949 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.346892 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.346949 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.346911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.347556 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.347538 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:04.347638 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.347564 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:04.347638 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.347575 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:04.347638 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.347603 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.396145 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.395992 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:04.402786 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.402765 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.402855 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.402790 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-181.us-west-2.compute.internal\": node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.447565 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.447542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/010a8efd81cfe8476c37b8d73ee870b2-config\") pod \"kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"010a8efd81cfe8476c37b8d73ee870b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.447656 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.447568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.447656 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.447584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.447656 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.447641 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.447656 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.447647 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/010a8efd81cfe8476c37b8d73ee870b2-config\") pod \"kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"010a8efd81cfe8476c37b8d73ee870b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.447656 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.447652 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.513377 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.513304 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.613139 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.613109 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.614174 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.614152 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.618432 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.618418 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:04.714964 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.714918 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.818213 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.818124 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.918799 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.918763 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:04.977882 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.977843 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 14:57:04.978445 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:04.978061 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 14:57:04.978445 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:04.978066 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae753dff0150c41c2b123ad91ad36b56-7d3113d1bc7bb9ad.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-181.us-west-2.compute.internal.18e44246f17af204\": read tcp 10.0.135.181:41928->3.151.193.16:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-181.us-west-2.compute.internal.18e44246f17af204 default 5171 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-181.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-181.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-181.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:57:04 +0000 UTC,LastTimestamp:2026-11-02 14:57:04.280887771 +0000 UTC m=+0.642713730,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-181.us-west-2.compute.internal,}" Nov 02 14:57:05.019298 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:05.019257 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:05.119535 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:05.119449 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:05.220240 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:05.220204 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:57:05.241538 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.241509 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:05.245266 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.245242 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:05.271517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.271482 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:52:04 +0000 UTC" deadline="2028-05-17 13:15:32.719365747 +0000 UTC" Nov 02 14:57:05.271517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.271515 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13486h18m27.447857343s" Nov 02 14:57:05.294024 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.294002 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:05.316192 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.316166 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:57:05.318398 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.318380 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:57:05.385176 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.385151 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:57:05.643839 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.643754 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:05.886067 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:05.886010 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010a8efd81cfe8476c37b8d73ee870b2.slice/crio-a5c71fadb1c1bc9e1fc6f904ccea9974225acd46a547316a4b537794092e5c12 WatchSource:0}: Error finding container a5c71fadb1c1bc9e1fc6f904ccea9974225acd46a547316a4b537794092e5c12: Status 404 returned error can't find the container with id a5c71fadb1c1bc9e1fc6f904ccea9974225acd46a547316a4b537794092e5c12 Nov 02 14:57:05.886383 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:05.886359 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78c4a25b0fa3f7dd2a10621ff614039.slice/crio-2fdade16c0d70ce01cd2a8280a9d25215113305d0f78e6caafa7938ae784b9f4 WatchSource:0}: Error finding container 2fdade16c0d70ce01cd2a8280a9d25215113305d0f78e6caafa7938ae784b9f4: Status 404 returned error can't find the container with id 2fdade16c0d70ce01cd2a8280a9d25215113305d0f78e6caafa7938ae784b9f4 Nov 02 14:57:05.892238 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:05.892213 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:57:06.020131 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.020091 2577 apiserver.go:52] "Watching apiserver" Nov 02 14:57:06.045605 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.045570 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 14:57:06.145324 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.145239 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:06.145324 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.145265 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 14:57:06.145625 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.145598 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-69x5r","openshift-network-operator/iptables-alerter-px579","openshift-network-diagnostics/network-check-target-l6p7c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv","openshift-image-registry/node-ca-jpfd7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-9j6h2","openshift-multus/network-metrics-daemon-sx5r5","openshift-ovn-kubernetes/ovnkube-node-mqvnb","kube-system/konnectivity-agent-4522z","kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-kcfxh","openshift-dns/node-resolver-2zz8m","openshift-multus/multus-frtdt"] Nov 02 14:57:06.148541 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.148522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.149909 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.149889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.151666 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.151647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:06.151757 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.151706 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:06.153171 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.153158 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.154750 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.154728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.156341 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.156322 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.157963 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.157944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.159956 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.159934 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.160068 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.160034 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:06.161449 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.161434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.163073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.163054 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:06.165154 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.165135 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:06.165233 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.165209 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:06.166883 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.166868 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.185390 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.185337 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"c78c4a25b0fa3f7dd2a10621ff614039","Type":"ContainerStarted","Data":"2fdade16c0d70ce01cd2a8280a9d25215113305d0f78e6caafa7938ae784b9f4"} Nov 02 14:57:06.186597 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.186570 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"010a8efd81cfe8476c37b8d73ee870b2","Type":"ContainerStarted","Data":"a5c71fadb1c1bc9e1fc6f904ccea9974225acd46a547316a4b537794092e5c12"} Nov 02 14:57:06.209363 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.209340 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q62fl\"" Nov 02 14:57:06.219867 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.219846 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.220272 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220256 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.220364 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220351 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 14:57:06.220421 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220384 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 14:57:06.220421 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220404 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.220524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220356 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.220524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220428 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.220524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220439 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 14:57:06.220524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220388 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.220524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220458 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.220524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220460 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.220524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220473 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 14:57:06.220524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.220494 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 14:57:06.247299 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.247278 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 14:57:06.257712 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.257685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-netns\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.257800 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.257720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-multus\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.257800 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.257749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cnibin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.257800 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.257774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cni-binary-copy\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.257930 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.257844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-host\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.257930 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.257911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.258028 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.257953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.258028 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.257977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-socket-dir-parent\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.258028 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.258149 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-registration-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.258149 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-device-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.258149 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-run\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.258149 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-cnibin\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.258149 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-systemd-units\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.258352 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-node-log\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.258352 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-log-socket\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.258352 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.258352 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-daemon-config\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.258352 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-systemd\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.258352 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlfss\" (UniqueName: \"kubernetes.io/projected/2dc3e10d-fde4-4b8a-97b7-e065848bed64-kube-api-access-xlfss\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.258352 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9695a92-acd9-41de-bfcf-13d149ecb730-tmp-dir\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.258352 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258335 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-env-overrides\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.258352 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-sys-fs\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258435 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795a4ecf-3ca0-4699-a06a-a051078f9fd1-host\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795a4ecf-3ca0-4699-a06a-a051078f9fd1-serviceca\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-bin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-binary-copy\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-hostroot\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-socket-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-etc-selinux\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-etc-kubernetes\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2616b3d7-11af-4008-99e7-44a5c575e85a-agent-certs\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:06.258744 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-modprobe-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-sys\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258799 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-bin\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsqwp\" (UniqueName: \"kubernetes.io/projected/795a4ecf-3ca0-4699-a06a-a051078f9fd1-kube-api-access-hsqwp\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-system-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0364fe3-6710-4de9-af3b-cd66989104f8-host-slash\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-tuned\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-tmp\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.258959 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-ovn\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhzc\" (UniqueName: \"kubernetes.io/projected/80cddaf6-6206-4424-bff1-953ac57d7c81-kube-api-access-6hhzc\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-conf-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q976l\" (UniqueName: \"kubernetes.io/projected/a0364fe3-6710-4de9-af3b-cd66989104f8-kube-api-access-q976l\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-script-lib\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-kubelet-config\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.259270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysconfig\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-system-cni-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-slash\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-k8s-cni-cncf-io\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0364fe3-6710-4de9-af3b-cd66989104f8-iptables-alerter-script\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-systemd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-config\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-multus-certs\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-kubelet\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-etc-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-netd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb969\" (UniqueName: \"kubernetes.io/projected/6c7d4401-1b82-4665-8838-9e05fac70e69-kube-api-access-nb969\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259432 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6kz\" (UniqueName: \"kubernetes.io/projected/9ac6fea3-ce51-474e-bb44-9b328a950fbe-kube-api-access-7b6kz\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2616b3d7-11af-4008-99e7-44a5c575e85a-konnectivity-ca\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259474 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-lib-modules\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.259731 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-var-lib-kubelet\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9695a92-acd9-41de-bfcf-13d149ecb730-hosts-file\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-var-lib-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259559 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-dbus\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-kubelet\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-kubernetes\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-conf\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mr5q\" (UniqueName: \"kubernetes.io/projected/8b6bfaad-2c57-4baa-be7d-f25715574792-kube-api-access-4mr5q\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-netns\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovn-node-metrics-cert\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsqxh\" (UniqueName: \"kubernetes.io/projected/e9695a92-acd9-41de-bfcf-13d149ecb730-kube-api-access-dsqxh\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.260202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.259833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jvst\" (UniqueName: \"kubernetes.io/projected/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-kube-api-access-6jvst\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.261521 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.261502 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 14:57:06.261566 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.261560 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g878g" Nov 02 14:57:06.317677 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.317653 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 14:57:06.318210 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.318187 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ct4hh\"" Nov 02 14:57:06.318210 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.318200 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 14:57:06.318403 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.318277 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b9vbw\"" Nov 02 14:57:06.318403 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.318283 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 14:57:06.318403 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.318294 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hnsn9\"" Nov 02 14:57:06.318403 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.318288 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dlx4j\"" Nov 02 14:57:06.318403 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.318340 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s2tzl\"" Nov 02 14:57:06.349147 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349123 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tsn9p\"" Nov 02 14:57:06.349270 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349159 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2tgnv\"" Nov 02 14:57:06.349532 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349515 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 14:57:06.349532 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349522 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.349532 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349532 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 14:57:06.349724 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349557 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.349724 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349557 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.349724 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349587 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.349724 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349595 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4crbl\"" Nov 02 14:57:06.349724 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349608 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.349724 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.349626 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 14:57:06.361013 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.360978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-device-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.361110 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-run\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.361110 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-cnibin\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.361110 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-systemd-units\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.361110 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-node-log\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.361110 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-log-socket\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361124 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-device-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361152 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-systemd-units\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361153 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-cnibin\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361194 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-node-log\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-daemon-config\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361246 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-log-socket\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-systemd\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlfss\" (UniqueName: \"kubernetes.io/projected/2dc3e10d-fde4-4b8a-97b7-e065848bed64-kube-api-access-xlfss\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361307 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-systemd\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361320 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9695a92-acd9-41de-bfcf-13d149ecb730-tmp-dir\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.361402 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-env-overrides\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-sys-fs\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795a4ecf-3ca0-4699-a06a-a051078f9fd1-host\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361623 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-sys-fs\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-run\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795a4ecf-3ca0-4699-a06a-a051078f9fd1-serviceca\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-bin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-binary-copy\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-hostroot\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361855 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-daemon-config\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361896 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795a4ecf-3ca0-4699-a06a-a051078f9fd1-host\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361913 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-bin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.361950 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361961 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-hostroot\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.362073 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.361867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-socket-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.362053 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs podName:6c7d4401-1b82-4665-8838-9e05fac70e69 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:06.862015691 +0000 UTC m=+3.223841655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs") pod "network-metrics-daemon-sx5r5" (UID: "6c7d4401-1b82-4665-8838-9e05fac70e69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-etc-selinux\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-etc-kubernetes\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362112 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2616b3d7-11af-4008-99e7-44a5c575e85a-agent-certs\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362075 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-socket-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362173 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-etc-kubernetes\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-modprobe-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362214 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-sys\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362155 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-etc-selinux\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-bin\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362215 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9695a92-acd9-41de-bfcf-13d149ecb730-tmp-dir\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsqwp\" (UniqueName: \"kubernetes.io/projected/795a4ecf-3ca0-4699-a06a-a051078f9fd1-kube-api-access-hsqwp\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-system-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362304 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-modprobe-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0364fe3-6710-4de9-af3b-cd66989104f8-host-slash\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.362874 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-tuned\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362390 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-sys\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.362392 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362438 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-system-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.362467 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret podName:ed84914e-49d4-4f14-877b-ca1b0cdff144 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:06.862447287 +0000 UTC m=+3.224273253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret") pod "global-pull-secret-syncer-69x5r" (UID: "ed84914e-49d4-4f14-877b-ca1b0cdff144") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362469 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-bin\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-tmp\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362509 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795a4ecf-3ca0-4699-a06a-a051078f9fd1-serviceca\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362545 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0364fe3-6710-4de9-af3b-cd66989104f8-host-slash\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362578 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-ovn\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362626 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-ovn\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362692 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362706 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhzc\" (UniqueName: \"kubernetes.io/projected/80cddaf6-6206-4424-bff1-953ac57d7c81-kube-api-access-6hhzc\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-conf-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.363704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q976l\" (UniqueName: \"kubernetes.io/projected/a0364fe3-6710-4de9-af3b-cd66989104f8-kube-api-access-q976l\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-script-lib\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-kubelet-config\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysconfig\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-system-cni-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-slash\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362846 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-conf-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.362915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-k8s-cni-cncf-io\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0364fe3-6710-4de9-af3b-cd66989104f8-iptables-alerter-script\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363061 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-kubelet-config\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-systemd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-slash\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-config\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-multus-certs\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-kubelet\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysconfig\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.364635 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363386 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-env-overrides\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363465 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-k8s-cni-cncf-io\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363511 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-system-cni-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-etc-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363625 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-netd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363653 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-systemd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb969\" (UniqueName: \"kubernetes.io/projected/6c7d4401-1b82-4665-8838-9e05fac70e69-kube-api-access-nb969\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-netd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363776 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-kubelet\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6kz\" (UniqueName: \"kubernetes.io/projected/9ac6fea3-ce51-474e-bb44-9b328a950fbe-kube-api-access-7b6kz\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2616b3d7-11af-4008-99e7-44a5c575e85a-konnectivity-ca\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363923 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-multus-certs\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-lib-modules\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-var-lib-kubelet\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-etc-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-binary-copy\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.365506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364049 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-var-lib-kubelet\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364076 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-config\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364089 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9695a92-acd9-41de-bfcf-13d149ecb730-hosts-file\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364080 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-lib-modules\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.363990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9695a92-acd9-41de-bfcf-13d149ecb730-hosts-file\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-var-lib-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-dbus\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-kubelet\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-dbus\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-kubernetes\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364290 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-var-lib-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-conf\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364404 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-kubelet\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mr5q\" (UniqueName: \"kubernetes.io/projected/8b6bfaad-2c57-4baa-be7d-f25715574792-kube-api-access-4mr5q\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364438 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-kubernetes\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364541 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-conf\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:06.366343 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-netns\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovn-node-metrics-cert\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364674 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2616b3d7-11af-4008-99e7-44a5c575e85a-konnectivity-ca\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364711 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364719 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-netns\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsqxh\" (UniqueName: \"kubernetes.io/projected/e9695a92-acd9-41de-bfcf-13d149ecb730-kube-api-access-dsqxh\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jvst\" (UniqueName: \"kubernetes.io/projected/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-kube-api-access-6jvst\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-netns\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.364973 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-netns\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-multus\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cnibin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365065 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cni-binary-copy\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365099 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-multus\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-host\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365148 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cnibin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.367091 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365212 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-host\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-socket-dir-parent\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-registration-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365433 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-registration-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365483 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365565 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-socket-dir-parent\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.365638 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cni-binary-copy\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.366394 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.367406 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.367525 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-tmp\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.367538 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-tuned\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.367646 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovn-node-metrics-cert\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.367699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.367698 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2616b3d7-11af-4008-99e7-44a5c575e85a-agent-certs\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:06.373647 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.373627 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.373719 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.373656 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 14:57:06.373776 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.373759 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 14:57:06.374226 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.374210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0364fe3-6710-4de9-af3b-cd66989104f8-iptables-alerter-script\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.384387 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.384364 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-script-lib\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.429014 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.428947 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlfss\" (UniqueName: \"kubernetes.io/projected/2dc3e10d-fde4-4b8a-97b7-e065848bed64-kube-api-access-xlfss\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.430681 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.430656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q976l\" (UniqueName: \"kubernetes.io/projected/a0364fe3-6710-4de9-af3b-cd66989104f8-kube-api-access-q976l\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.436451 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.436420 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:06.436451 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.436452 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:06.436628 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.436470 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gdzxj for pod openshift-network-diagnostics/network-check-target-l6p7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:06.436628 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.436561 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj podName:8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:06.936533437 +0000 UTC m=+3.298359404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gdzxj" (UniqueName: "kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj") pod "network-check-target-l6p7c" (UID: "8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:06.437634 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.437544 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsqwp\" (UniqueName: \"kubernetes.io/projected/795a4ecf-3ca0-4699-a06a-a051078f9fd1-kube-api-access-hsqwp\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.441015 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.438373 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsqxh\" (UniqueName: \"kubernetes.io/projected/e9695a92-acd9-41de-bfcf-13d149ecb730-kube-api-access-dsqxh\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.441015 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.440631 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhzc\" (UniqueName: \"kubernetes.io/projected/80cddaf6-6206-4424-bff1-953ac57d7c81-kube-api-access-6hhzc\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.441015 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.440636 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6kz\" (UniqueName: \"kubernetes.io/projected/9ac6fea3-ce51-474e-bb44-9b328a950fbe-kube-api-access-7b6kz\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.441015 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.440767 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jvst\" (UniqueName: \"kubernetes.io/projected/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-kube-api-access-6jvst\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.441597 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.441575 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mr5q\" (UniqueName: \"kubernetes.io/projected/8b6bfaad-2c57-4baa-be7d-f25715574792-kube-api-access-4mr5q\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.441967 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.441948 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb969\" (UniqueName: \"kubernetes.io/projected/6c7d4401-1b82-4665-8838-9e05fac70e69-kube-api-access-nb969\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:06.468004 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.467968 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:57:06.469138 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.469121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:57:06.477169 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.477144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:57:06.479095 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.479070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:57:06.481431 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:06.481405 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6bfaad_2c57_4baa_be7d_f25715574792.slice/crio-064e024e7af9f230b540bb3051a055b92cd7bb0148e04d749780600bd7be8bf4 WatchSource:0}: Error finding container 064e024e7af9f230b540bb3051a055b92cd7bb0148e04d749780600bd7be8bf4: Status 404 returned error can't find the container with id 064e024e7af9f230b540bb3051a055b92cd7bb0148e04d749780600bd7be8bf4 Nov 02 14:57:06.483262 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:06.483242 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9695a92_acd9_41de_bfcf_13d149ecb730.slice/crio-6c7a75b887a2f15f14e1e7f1b82ba9dcafda77d8ff0717a53f87d05aa32cf2e5 WatchSource:0}: Error finding container 6c7a75b887a2f15f14e1e7f1b82ba9dcafda77d8ff0717a53f87d05aa32cf2e5: Status 404 returned error can't find the container with id 6c7a75b887a2f15f14e1e7f1b82ba9dcafda77d8ff0717a53f87d05aa32cf2e5 Nov 02 14:57:06.488644 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.488620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-frtdt" Nov 02 14:57:06.489499 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:06.489468 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cddaf6_6206_4424_bff1_953ac57d7c81.slice/crio-46ea21093ca88ee225361b61733b6c3e59e045dcaf8bf3d6de3d3d832491fcac WatchSource:0}: Error finding container 46ea21093ca88ee225361b61733b6c3e59e045dcaf8bf3d6de3d3d832491fcac: Status 404 returned error can't find the container with id 46ea21093ca88ee225361b61733b6c3e59e045dcaf8bf3d6de3d3d832491fcac Nov 02 14:57:06.490407 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:06.490381 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795a4ecf_3ca0_4699_a06a_a051078f9fd1.slice/crio-4cd2d0008bd4daadc62312915a67f350ebd6850b885725020e29208c6219deee WatchSource:0}: Error finding container 4cd2d0008bd4daadc62312915a67f350ebd6850b885725020e29208c6219deee: Status 404 returned error can't find the container with id 4cd2d0008bd4daadc62312915a67f350ebd6850b885725020e29208c6219deee Nov 02 14:57:06.496948 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:06.496930 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac6fea3_ce51_474e_bb44_9b328a950fbe.slice/crio-33caa47232f121ab61096a4b20f93a65e9605f8dd64cc61870fe44857d734578 WatchSource:0}: Error finding container 33caa47232f121ab61096a4b20f93a65e9605f8dd64cc61870fe44857d734578: Status 404 returned error can't find the container with id 33caa47232f121ab61096a4b20f93a65e9605f8dd64cc61870fe44857d734578 Nov 02 14:57:06.503201 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.503181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:57:06.504320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.504302 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:06.506603 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.506575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:06.511579 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.511558 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:57:06.513494 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:06.513399 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0364fe3_6710_4de9_af3b_cd66989104f8.slice/crio-cc5cf81c7d5fe600c530c87d583076f795c8fd1e528987c6da619cc71eb113ac WatchSource:0}: Error finding container cc5cf81c7d5fe600c530c87d583076f795c8fd1e528987c6da619cc71eb113ac: Status 404 returned error can't find the container with id cc5cf81c7d5fe600c530c87d583076f795c8fd1e528987c6da619cc71eb113ac Nov 02 14:57:06.516196 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:06.516163 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eff5e30_0f92_4736_a63f_c2a02ceb5a47.slice/crio-e8fbbd7dcba1878386c62ef3067fb92cd8b2be038bb29181b570d5610df31e48 WatchSource:0}: Error finding container e8fbbd7dcba1878386c62ef3067fb92cd8b2be038bb29181b570d5610df31e48: Status 404 returned error can't find the container with id e8fbbd7dcba1878386c62ef3067fb92cd8b2be038bb29181b570d5610df31e48 Nov 02 14:57:06.517515 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:06.517493 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2616b3d7_11af_4008_99e7_44a5c575e85a.slice/crio-a87a11243e7dcbadb85ed7f01766cd29da43e41dba0bb095536e0c3fff77d794 WatchSource:0}: Error finding container a87a11243e7dcbadb85ed7f01766cd29da43e41dba0bb095536e0c3fff77d794: Status 404 returned error can't find the container with id a87a11243e7dcbadb85ed7f01766cd29da43e41dba0bb095536e0c3fff77d794 Nov 02 14:57:06.521823 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:06.521794 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc3e10d_fde4_4b8a_97b7_e065848bed64.slice/crio-f45cbd408acf3de88780085c6ca5ed14c302ea08af88e30b29fd487c642d32a3 WatchSource:0}: Error finding container f45cbd408acf3de88780085c6ca5ed14c302ea08af88e30b29fd487c642d32a3: Status 404 returned error can't find the container with id f45cbd408acf3de88780085c6ca5ed14c302ea08af88e30b29fd487c642d32a3 Nov 02 14:57:06.869715 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.869628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:06.869715 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.869676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:06.869883 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.869775 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:06.869883 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.869775 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:06.869883 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.869842 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret podName:ed84914e-49d4-4f14-877b-ca1b0cdff144 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:07.86982694 +0000 UTC m=+4.231652905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret") pod "global-pull-secret-syncer-69x5r" (UID: "ed84914e-49d4-4f14-877b-ca1b0cdff144") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:06.869981 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.869889 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs podName:6c7d4401-1b82-4665-8838-9e05fac70e69 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:07.869878149 +0000 UTC m=+4.231704100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs") pod "network-metrics-daemon-sx5r5" (UID: "6c7d4401-1b82-4665-8838-9e05fac70e69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:06.970399 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:06.970361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:06.970564 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.970519 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:06.970564 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.970540 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:06.970564 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.970551 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gdzxj for pod openshift-network-diagnostics/network-check-target-l6p7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:06.970682 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:06.970601 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj podName:8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:07.970586954 +0000 UTC m=+4.332412916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdzxj" (UniqueName: "kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj") pod "network-check-target-l6p7c" (UID: "8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:07.191389 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.191352 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4522z" event={"ID":"2616b3d7-11af-4008-99e7-44a5c575e85a","Type":"ContainerStarted","Data":"a87a11243e7dcbadb85ed7f01766cd29da43e41dba0bb095536e0c3fff77d794"} Nov 02 14:57:07.193157 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.193121 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" event={"ID":"7eff5e30-0f92-4736-a63f-c2a02ceb5a47","Type":"ContainerStarted","Data":"e8fbbd7dcba1878386c62ef3067fb92cd8b2be038bb29181b570d5610df31e48"} Nov 02 14:57:07.195327 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.195272 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-px579" event={"ID":"a0364fe3-6710-4de9-af3b-cd66989104f8","Type":"ContainerStarted","Data":"cc5cf81c7d5fe600c530c87d583076f795c8fd1e528987c6da619cc71eb113ac"} Nov 02 14:57:07.196848 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.196819 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpfd7" event={"ID":"795a4ecf-3ca0-4699-a06a-a051078f9fd1","Type":"ContainerStarted","Data":"4cd2d0008bd4daadc62312915a67f350ebd6850b885725020e29208c6219deee"} Nov 02 14:57:07.198400 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.198371 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" event={"ID":"2dc3e10d-fde4-4b8a-97b7-e065848bed64","Type":"ContainerStarted","Data":"f45cbd408acf3de88780085c6ca5ed14c302ea08af88e30b29fd487c642d32a3"} Nov 02 14:57:07.200396 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.200372 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-frtdt" event={"ID":"9ac6fea3-ce51-474e-bb44-9b328a950fbe","Type":"ContainerStarted","Data":"33caa47232f121ab61096a4b20f93a65e9605f8dd64cc61870fe44857d734578"} Nov 02 14:57:07.201829 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.201773 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" event={"ID":"80cddaf6-6206-4424-bff1-953ac57d7c81","Type":"ContainerStarted","Data":"46ea21093ca88ee225361b61733b6c3e59e045dcaf8bf3d6de3d3d832491fcac"} Nov 02 14:57:07.203447 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.203417 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2zz8m" event={"ID":"e9695a92-acd9-41de-bfcf-13d149ecb730","Type":"ContainerStarted","Data":"6c7a75b887a2f15f14e1e7f1b82ba9dcafda77d8ff0717a53f87d05aa32cf2e5"} Nov 02 14:57:07.205263 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.205231 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" event={"ID":"8b6bfaad-2c57-4baa-be7d-f25715574792","Type":"ContainerStarted","Data":"064e024e7af9f230b540bb3051a055b92cd7bb0148e04d749780600bd7be8bf4"} Nov 02 14:57:07.263541 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.263493 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:52:06 +0000 UTC" deadline="2028-06-22 23:34:31.280368277 +0000 UTC" Nov 02 14:57:07.263541 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.263531 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14360h37m24.016844252s" Nov 02 14:57:07.881598 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.881557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:07.881780 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.881617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:07.881852 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:07.881827 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:07.881920 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:07.881893 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs podName:6c7d4401-1b82-4665-8838-9e05fac70e69 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:09.88187256 +0000 UTC m=+6.243698534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs") pod "network-metrics-daemon-sx5r5" (UID: "6c7d4401-1b82-4665-8838-9e05fac70e69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:07.882076 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:07.881964 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:07.882076 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:07.882000 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret podName:ed84914e-49d4-4f14-877b-ca1b0cdff144 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:09.881988819 +0000 UTC m=+6.243814773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret") pod "global-pull-secret-syncer-69x5r" (UID: "ed84914e-49d4-4f14-877b-ca1b0cdff144") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:07.982803 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:07.982761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:07.983086 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:07.983051 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:07.983086 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:07.983078 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:07.983237 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:07.983094 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gdzxj for pod openshift-network-diagnostics/network-check-target-l6p7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:07.983294 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:07.983254 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj podName:8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:09.983200757 +0000 UTC m=+6.345026727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdzxj" (UniqueName: "kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj") pod "network-check-target-l6p7c" (UID: "8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:08.180226 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:08.180139 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:08.180389 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:08.180302 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:08.180468 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:08.180416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:08.180541 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:08.180519 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:08.183281 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:08.183251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:08.183424 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:08.183368 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:08.263899 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:08.263780 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:52:06 +0000 UTC" deadline="2028-05-30 09:16:49.794157014 +0000 UTC" Nov 02 14:57:08.263899 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:08.263900 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13794h19m41.530263325s" Nov 02 14:57:09.898533 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:09.898488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:09.899149 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:09.898538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:09.899149 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:09.898698 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:09.899149 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:09.898763 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs podName:6c7d4401-1b82-4665-8838-9e05fac70e69 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:13.898743996 +0000 UTC m=+10.260569958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs") pod "network-metrics-daemon-sx5r5" (UID: "6c7d4401-1b82-4665-8838-9e05fac70e69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:09.899342 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:09.899178 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:09.899342 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:09.899234 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret podName:ed84914e-49d4-4f14-877b-ca1b0cdff144 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:13.899217941 +0000 UTC m=+10.261043891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret") pod "global-pull-secret-syncer-69x5r" (UID: "ed84914e-49d4-4f14-877b-ca1b0cdff144") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:09.999217 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:09.999179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:09.999376 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:09.999341 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:09.999376 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:09.999361 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:09.999376 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:09.999373 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gdzxj for pod openshift-network-diagnostics/network-check-target-l6p7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:09.999544 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:09.999432 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj podName:8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:13.999412726 +0000 UTC m=+10.361238686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdzxj" (UniqueName: "kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj") pod "network-check-target-l6p7c" (UID: "8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:10.182601 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:10.182564 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:10.182788 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:10.182752 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:10.182861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:10.182847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:10.182961 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:10.182939 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:10.183047 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:10.183021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:10.183131 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:10.183109 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:11.258547 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:11.258304 2577 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="76c468396751e4af18f55c5d41b3001a7d138d3c342e12bc44377b5cd1aff381" exitCode=0 Nov 02 14:57:11.259025 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:11.258500 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" event={"ID":"8b6bfaad-2c57-4baa-be7d-f25715574792","Type":"ContainerDied","Data":"76c468396751e4af18f55c5d41b3001a7d138d3c342e12bc44377b5cd1aff381"} Nov 02 14:57:11.263693 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:11.263665 2577 generic.go:356] "Generic (PLEG): container finished" podID="c78c4a25b0fa3f7dd2a10621ff614039" containerID="339924069d77259f5b31ee96dfb63037399565cff30143f35eb75d90fc4e8c29" exitCode=0 Nov 02 14:57:11.263798 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:11.263761 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"c78c4a25b0fa3f7dd2a10621ff614039","Type":"ContainerDied","Data":"339924069d77259f5b31ee96dfb63037399565cff30143f35eb75d90fc4e8c29"} Nov 02 14:57:11.273205 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:11.273167 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"010a8efd81cfe8476c37b8d73ee870b2","Type":"ContainerStarted","Data":"cdb93fdbc918aabcf8f2767408d20dd1ca82bd837a61effa00cc9826bf752bae"} Nov 02 14:57:11.720660 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:11.720548 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" podStartSLOduration=6.720526985 podStartE2EDuration="6.720526985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:11.719665809 +0000 UTC m=+8.081491775" watchObservedRunningTime="2026-11-02 14:57:11.720526985 +0000 UTC m=+8.082352953" Nov 02 14:57:12.181772 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:12.181739 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:12.181914 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:12.181838 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:12.182212 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:12.182193 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:12.182294 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:12.182279 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:12.182339 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:12.182325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:12.182399 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:12.182380 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:12.276573 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:12.276537 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"c78c4a25b0fa3f7dd2a10621ff614039","Type":"ContainerStarted","Data":"6d16f7536cf5be2f51d91bc605d80ff9c791c1a35961eaa50f4a58ad4d249044"} Nov 02 14:57:12.284872 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:12.284845 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4522z" event={"ID":"2616b3d7-11af-4008-99e7-44a5c575e85a","Type":"ContainerStarted","Data":"455293518d16a06f9f2adbea5c270480565731dc3f30e8db8025249d79d12b49"} Nov 02 14:57:12.514408 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:12.514345 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" podStartSLOduration=7.514322113 podStartE2EDuration="7.514322113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:12.395389424 +0000 UTC m=+8.757215392" watchObservedRunningTime="2026-11-02 14:57:12.514322113 +0000 UTC m=+8.876148081" Nov 02 14:57:13.129771 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:13.129739 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:13.130775 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:13.130571 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:13.254247 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:13.254204 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4522z" podStartSLOduration=4.574859008 podStartE2EDuration="9.254188878s" totalImagesPullingTime="4.67932987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:12.515268578 +0000 UTC m=+8.877094549" watchObservedRunningTime="2026-11-02 14:57:13.254188878 +0000 UTC m=+9.616014844" Nov 02 14:57:13.290531 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:13.290503 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpfd7" event={"ID":"795a4ecf-3ca0-4699-a06a-a051078f9fd1","Type":"ContainerStarted","Data":"578156e13efb62d88024da6ef417d794a52d383cdb5e1254cac3a6ccdb6a5a2f"} Nov 02 14:57:13.930506 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:13.930461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:13.930666 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:13.930521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:13.930711 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:13.930664 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:13.930740 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:13.930729 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs podName:6c7d4401-1b82-4665-8838-9e05fac70e69 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:21.930708507 +0000 UTC m=+18.292534494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs") pod "network-metrics-daemon-sx5r5" (UID: "6c7d4401-1b82-4665-8838-9e05fac70e69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:13.931092 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:13.931072 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:13.931188 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:13.931118 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret podName:ed84914e-49d4-4f14-877b-ca1b0cdff144 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:21.93110828 +0000 UTC m=+18.292934239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret") pod "global-pull-secret-syncer-69x5r" (UID: "ed84914e-49d4-4f14-877b-ca1b0cdff144") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:14.031490 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:14.031451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:14.031647 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:14.031610 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:14.031647 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:14.031629 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:14.031647 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:14.031641 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gdzxj for pod openshift-network-diagnostics/network-check-target-l6p7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:14.031812 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:14.031699 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj podName:8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.03168198 +0000 UTC m=+18.393507952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdzxj" (UniqueName: "kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj") pod "network-check-target-l6p7c" (UID: "8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:14.182041 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:14.181438 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:14.182041 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:14.181549 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:14.182041 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:14.181684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:14.182041 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:14.181790 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:14.182041 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:14.181867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:14.182041 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:14.181952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:14.298307 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:14.298262 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:14.298779 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:14.298753 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" event={"ID":"80cddaf6-6206-4424-bff1-953ac57d7c81","Type":"ContainerStarted","Data":"27449c2ad7b6b8a7a743881c973d0a5fb2c7586a40be157b1f0b76f1cba38506"} Nov 02 14:57:15.306150 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:15.304549 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-frtdt" event={"ID":"9ac6fea3-ce51-474e-bb44-9b328a950fbe","Type":"ContainerStarted","Data":"c7da90f928c65b3d699b142a77bb3bc15c6995ddd5260ba23414af67b99b7d3d"} Nov 02 14:57:15.367312 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:15.367265 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 14:57:15.425761 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:15.425718 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jpfd7" podStartSLOduration=5.70474564 podStartE2EDuration="11.425705011s" totalImagesPullingTime="5.720959371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:13.40850563 +0000 UTC m=+9.770331598" watchObservedRunningTime="2026-11-02 14:57:15.425705011 +0000 UTC m=+11.787530977" Nov 02 14:57:15.426388 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:15.426342 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-frtdt" podStartSLOduration=2.857858929 podStartE2EDuration="11.426328475s" totalImagesPullingTime="8.568469546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:15.425362879 +0000 UTC m=+11.787188846" watchObservedRunningTime="2026-11-02 14:57:15.426328475 +0000 UTC m=+11.788154444" Nov 02 14:57:15.608734 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:15.608643 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:15.793657 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:15.793614 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:15.984908 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:15.984741 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:16.179604 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.179565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:16.179772 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:16.179704 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:16.180137 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.180122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:16.180243 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:16.180226 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:16.180320 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.180297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:16.180428 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:16.180410 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:16.251751 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.251635 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:57:15.367283495Z","UUID":"e9b78f7a-6839-4d84-b005-d4eefa200bbb","Handler":null,"Name":"","Endpoint":""} Nov 02 14:57:16.253899 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.253481 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 14:57:16.253899 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.253509 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 14:57:16.308990 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.308932 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" event={"ID":"80cddaf6-6206-4424-bff1-953ac57d7c81","Type":"ContainerStarted","Data":"94db2fffa0f5010743c3faeef2b47f963bf4f4a8f088548010583cd13c428709"} Nov 02 14:57:16.311153 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.311092 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2zz8m" event={"ID":"e9695a92-acd9-41de-bfcf-13d149ecb730","Type":"ContainerStarted","Data":"f2df47cf6a45355891cfb1ae13ad0382b12a3d5914376e1b7e951d6f7fda177f"} Nov 02 14:57:16.313089 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.313063 2577 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="8cae6207f58df68ca1ffe878cb2943b027bdda6ade3e7cb023b8e4207fb64186" exitCode=0 Nov 02 14:57:16.313213 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.313174 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" event={"ID":"8b6bfaad-2c57-4baa-be7d-f25715574792","Type":"ContainerDied","Data":"8cae6207f58df68ca1ffe878cb2943b027bdda6ade3e7cb023b8e4207fb64186"} Nov 02 14:57:16.433965 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.433853 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2zz8m" podStartSLOduration=3.404423096 podStartE2EDuration="12.433839455s" totalImagesPullingTime="9.029416359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:16.433166951 +0000 UTC m=+12.794992924" watchObservedRunningTime="2026-11-02 14:57:16.433839455 +0000 UTC m=+12.795665421" Nov 02 14:57:16.731597 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.731518 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:16.881494 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:16.881457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:17.057383 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:17.057312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:17.317524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:17.317435 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-px579" event={"ID":"a0364fe3-6710-4de9-af3b-cd66989104f8","Type":"ContainerStarted","Data":"b9aae27f84e9288a88b9de3f9d43e0291fc699e9cc2e39cc8ed173f829474a39"} Nov 02 14:57:17.434716 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:17.434657 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-px579" podStartSLOduration=4.569727959 podStartE2EDuration="13.434636415s" totalImagesPullingTime="8.864908456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:17.434431377 +0000 UTC m=+13.796257343" watchObservedRunningTime="2026-11-02 14:57:17.434636415 +0000 UTC m=+13.796462385" Nov 02 14:57:17.538278 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:17.538241 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:17.538461 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:17.538391 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:17.538844 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:17.538817 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4522z" Nov 02 14:57:17.737171 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:17.737138 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:18.179807 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.179777 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:18.179920 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.179778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:18.179920 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.179907 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:18.180050 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:18.179900 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:18.180050 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:18.179952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:18.180142 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:18.180081 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:18.322489 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.322402 2577 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="b45c9b6392f790d61d85d061b1958b79183cc32cf007787f5b0859647fa3907a" exitCode=0 Nov 02 14:57:18.323175 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.322495 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" event={"ID":"8b6bfaad-2c57-4baa-be7d-f25715574792","Type":"ContainerDied","Data":"b45c9b6392f790d61d85d061b1958b79183cc32cf007787f5b0859647fa3907a"} Nov 02 14:57:18.324628 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.324552 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" event={"ID":"2dc3e10d-fde4-4b8a-97b7-e065848bed64","Type":"ContainerStarted","Data":"ad4238d1da706f27a8b3039a7f163ab236490754909a21ad720988d145ce87fd"} Nov 02 14:57:18.326938 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.326912 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" event={"ID":"80cddaf6-6206-4424-bff1-953ac57d7c81","Type":"ContainerStarted","Data":"3b83e4cf76fb20ccb909ccff7011eaa5b8956b43d0b2a985cfe8c72857f7ad4d"} Nov 02 14:57:18.596077 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.595965 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" podStartSLOduration=3.822612022 podStartE2EDuration="14.595952639s" totalImagesPullingTime="10.773340617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:18.595379985 +0000 UTC m=+14.957205953" watchObservedRunningTime="2026-11-02 14:57:18.595952639 +0000 UTC m=+14.957778640" Nov 02 14:57:18.756709 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.756678 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:18.901808 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:18.901771 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:20.180369 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:20.180335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:20.180920 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:20.180335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:20.180920 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:20.180480 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:20.180920 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:20.180564 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:20.180920 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:20.180619 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:20.180920 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:20.180698 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:20.336059 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:20.335250 2577 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="8a8c2c28a165485cd564902af75529b29414e3d8fb0943cbcfb8bfbc082dbaa6" exitCode=0 Nov 02 14:57:20.336059 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:20.335387 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" event={"ID":"8b6bfaad-2c57-4baa-be7d-f25715574792","Type":"ContainerDied","Data":"8a8c2c28a165485cd564902af75529b29414e3d8fb0943cbcfb8bfbc082dbaa6"} Nov 02 14:57:20.527685 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:20.527635 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" podStartSLOduration=5.374131619 podStartE2EDuration="16.527619625s" totalImagesPullingTime="11.153488006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:18.718371904 +0000 UTC m=+15.080197870" watchObservedRunningTime="2026-11-02 14:57:20.527619625 +0000 UTC m=+16.889445614" Nov 02 14:57:21.856986 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:21.856930 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:21.994202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:21.994164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:21.994202 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:21.994208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:21.994401 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:21.994326 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:21.994401 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:21.994345 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:21.994474 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:21.994406 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret podName:ed84914e-49d4-4f14-877b-ca1b0cdff144 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:37.994384943 +0000 UTC m=+34.356210895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret") pod "global-pull-secret-syncer-69x5r" (UID: "ed84914e-49d4-4f14-877b-ca1b0cdff144") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:21.994474 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:21.994428 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs podName:6c7d4401-1b82-4665-8838-9e05fac70e69 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:37.994418578 +0000 UTC m=+34.356244540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs") pod "network-metrics-daemon-sx5r5" (UID: "6c7d4401-1b82-4665-8838-9e05fac70e69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:22.002658 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.002629 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:22.094777 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.094747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:22.094918 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:22.094901 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:22.094988 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:22.094924 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:22.094988 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:22.094934 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gdzxj for pod openshift-network-diagnostics/network-check-target-l6p7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:22.095091 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:22.094994 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj podName:8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:38.094961114 +0000 UTC m=+34.456787077 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdzxj" (UniqueName: "kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj") pod "network-check-target-l6p7c" (UID: "8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:22.150512 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.150478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:22.179831 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.179808 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:22.179961 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.179857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:22.179961 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.179868 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:22.180086 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:22.180006 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:22.180138 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:22.180084 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:22.180185 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:22.180151 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:22.304366 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.304195 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:22.342493 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.342466 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" event={"ID":"7eff5e30-0f92-4736-a63f-c2a02ceb5a47","Type":"ContainerStarted","Data":"c3e42082a41c68703a6cda3d4dcfbd07f717b8e709cf8e0d62b19293da68ca9d"} Nov 02 14:57:22.342586 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.342552 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" event={"ID":"7eff5e30-0f92-4736-a63f-c2a02ceb5a47","Type":"ContainerStarted","Data":"c0e86d7d9fefc9e4557a73a0b3fd24ff2aafb448bd3b01d7f775422e39e8d6c0"} Nov 02 14:57:22.342652 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.342604 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" event={"ID":"7eff5e30-0f92-4736-a63f-c2a02ceb5a47","Type":"ContainerStarted","Data":"320b3b9cbd804ccd6fca49087d2c0127ec7390a8fc179246724cd288b814b16f"} Nov 02 14:57:22.342703 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.342651 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" event={"ID":"7eff5e30-0f92-4736-a63f-c2a02ceb5a47","Type":"ContainerStarted","Data":"d157e9a2706d11ad6110d388d3921c2b3905e9264a0a2f2800d8c3f323afd320"} Nov 02 14:57:22.491916 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.491880 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:22.647672 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.647603 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:22.797864 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.797830 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:22.947513 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:22.947435 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:23.100687 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:23.100656 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:23.347592 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:23.347512 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" event={"ID":"7eff5e30-0f92-4736-a63f-c2a02ceb5a47","Type":"ContainerStarted","Data":"cbe5c40d0e695d640100e98934488cd1e1587804c05d6e5294aa97bfe6183c9a"} Nov 02 14:57:23.347592 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:23.347580 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" event={"ID":"7eff5e30-0f92-4736-a63f-c2a02ceb5a47","Type":"ContainerStarted","Data":"fbc1d292817833938a99e3f2098dea37d964be4268b59bbade30834bc5ebb79c"} Nov 02 14:57:23.350067 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:23.350040 2577 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="70aa349b8c2612b698c21f3ec39b0d33d99e8cf6a4c234bd9c9a5827cc5af753" exitCode=0 Nov 02 14:57:23.350141 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:23.350075 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" event={"ID":"8b6bfaad-2c57-4baa-be7d-f25715574792","Type":"ContainerDied","Data":"70aa349b8c2612b698c21f3ec39b0d33d99e8cf6a4c234bd9c9a5827cc5af753"} Nov 02 14:57:24.180727 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:24.180691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:24.181183 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:24.180807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:24.181183 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:24.180831 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:24.181183 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:24.180855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:24.181183 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:24.180940 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:24.181183 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:24.181046 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:24.207718 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:24.207687 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:24.354228 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:24.354188 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:24.355195 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:24.355167 2577 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="80f80e0fc7255358158b5271c185bfe278fbe50e811658b33837de57ec7976df" exitCode=0 Nov 02 14:57:24.355289 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:24.355234 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" event={"ID":"8b6bfaad-2c57-4baa-be7d-f25715574792","Type":"ContainerDied","Data":"80f80e0fc7255358158b5271c185bfe278fbe50e811658b33837de57ec7976df"} Nov 02 14:57:24.919077 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:24.919044 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:25.360184 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:25.360100 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" event={"ID":"7eff5e30-0f92-4736-a63f-c2a02ceb5a47","Type":"ContainerStarted","Data":"f3f3f9b567d6d362e2e3ddf1a4a442be6ec734e242688f3413b65f0b6a200aee"} Nov 02 14:57:25.363026 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:25.363003 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" event={"ID":"8b6bfaad-2c57-4baa-be7d-f25715574792","Type":"ContainerStarted","Data":"6ad047779ebd36fba54c5e0c3c85a6b6fde63dadeff39411a067df9d6043054b"} Nov 02 14:57:25.489081 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:25.489030 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9j6h2" podStartSLOduration=7.92546471 podStartE2EDuration="21.489015864s" totalImagesPullingTime="13.563551154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:25.488746697 +0000 UTC m=+21.850572667" watchObservedRunningTime="2026-11-02 14:57:25.489015864 +0000 UTC m=+21.850841831" Nov 02 14:57:26.180321 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:26.180280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:26.180494 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:26.180348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:26.180494 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:26.180353 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:26.180494 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:26.180453 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:26.180586 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:26.180564 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:26.180662 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:26.180643 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:26.283934 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:26.283901 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:26.421081 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:26.421055 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2zz8m_e9695a92-acd9-41de-bfcf-13d149ecb730/dns-node-resolver/0.log" Nov 02 14:57:26.861811 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:26.861710 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.009706 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.009677 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.191554 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.191528 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.372379 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.372350 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" event={"ID":"7eff5e30-0f92-4736-a63f-c2a02ceb5a47","Type":"ContainerStarted","Data":"9b138fd9acf83fc356924aa9459bef93137195504184ae34e2eb1b43d2c73690"} Nov 02 14:57:27.372644 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.372624 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:27.388215 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.388195 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:27.441670 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.441602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jpfd7_795a4ecf-3ca0-4699-a06a-a051078f9fd1/node-ca/0.log" Nov 02 14:57:27.540080 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.540046 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.670623 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.670559 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" podStartSLOduration=8.099626694 podStartE2EDuration="23.670540932s" totalImagesPullingTime="15.570914238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:27.669735511 +0000 UTC m=+24.031561479" watchObservedRunningTime="2026-11-02 14:57:27.670540932 +0000 UTC m=+24.032366901" Nov 02 14:57:27.697680 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.697610 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:27.846273 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:27.846239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:28.179777 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.179750 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:28.179915 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.179807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:28.179915 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.179816 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:28.180030 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:28.179904 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:28.180030 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:28.179986 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:28.180103 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:28.180038 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:28.369959 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.369930 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:28.374355 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.374336 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:28.374730 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.374718 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:28.389104 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.389084 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:28.548934 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.548860 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:28.745991 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.745948 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:28.945712 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:28.945683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:29.155721 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:29.155688 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:29.376445 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:29.376418 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:30.179631 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:30.179600 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:30.180103 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:30.179598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:30.180103 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:30.179716 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:30.180103 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:30.179598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:30.180103 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:30.179779 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:30.180103 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:30.179843 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:30.345825 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:30.345778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:30.378324 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:30.378298 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:30.503624 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:30.503538 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:30.653517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:30.653491 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:30.827094 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:30.827015 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:30.979777 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:30.979729 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:31.145993 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:31.145953 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:32.179999 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:32.179950 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:32.179999 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:32.179986 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:32.180481 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:32.180074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:32.180481 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:32.180091 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:32.180481 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:32.180155 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:32.180481 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:32.180212 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:32.346548 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:32.346520 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:32.547588 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:32.547511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:32.745209 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:32.745174 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:32.962059 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:32.962029 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:33.159719 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:33.159688 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:33.222540 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:33.222450 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:57:33.223017 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:33.222683 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:33.232789 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:33.232754 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" podUID="7eff5e30-0f92-4736-a63f-c2a02ceb5a47" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 14:57:33.241169 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:33.241141 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" podUID="7eff5e30-0f92-4736-a63f-c2a02ceb5a47" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 14:57:33.747512 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:33.747480 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:34.153068 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:34.153041 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:34.180393 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:34.180357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:34.180555 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:34.180443 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:34.180555 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:34.180541 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:34.180677 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:34.180665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:34.180736 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:34.180666 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:34.180792 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:34.180759 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:34.302286 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:34.302255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:34.545186 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:34.545107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:34.745340 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:34.745307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:34.942837 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:34.942797 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:35.368164 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:35.368081 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:36.143591 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:36.143564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:36.179675 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:36.179649 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:36.179806 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:36.179663 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:36.179806 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:36.179748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:36.179806 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:36.179776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:36.179899 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:36.179863 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:36.179951 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:36.179936 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:36.544795 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:36.544712 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:36.702955 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:36.702923 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:36.859773 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:36.859689 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:37.023299 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:37.023254 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:37.172591 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:37.172563 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:38.006388 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:38.006349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:38.006388 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:38.006389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:38.006850 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.006520 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:38.006850 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.006581 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret podName:ed84914e-49d4-4f14-877b-ca1b0cdff144 nodeName:}" failed. No retries permitted until 2026-11-02 14:58:10.006566254 +0000 UTC m=+66.368392203 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret") pod "global-pull-secret-syncer-69x5r" (UID: "ed84914e-49d4-4f14-877b-ca1b0cdff144") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:38.006850 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.006520 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:38.006850 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.006642 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs podName:6c7d4401-1b82-4665-8838-9e05fac70e69 nodeName:}" failed. No retries permitted until 2026-11-02 14:58:10.006629203 +0000 UTC m=+66.368455161 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs") pod "network-metrics-daemon-sx5r5" (UID: "6c7d4401-1b82-4665-8838-9e05fac70e69") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:38.107602 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:38.107564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:38.107771 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.107720 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:38.107771 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.107741 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:38.107771 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.107751 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gdzxj for pod openshift-network-diagnostics/network-check-target-l6p7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:38.107864 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.107801 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj podName:8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4 nodeName:}" failed. No retries permitted until 2026-11-02 14:58:10.10778708 +0000 UTC m=+66.469613042 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdzxj" (UniqueName: "kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj") pod "network-check-target-l6p7c" (UID: "8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:38.180174 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:38.180139 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:38.180353 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:38.180139 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:38.180353 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.180246 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:38.180353 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:38.180260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:38.180353 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.180312 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:38.180482 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:38.180407 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:39.442866 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:39.442837 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:39.598821 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:39.598787 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:39.745765 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:39.745697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:40.179689 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:40.179658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:40.179841 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:40.179658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:40.179841 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:40.179767 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:40.179913 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:40.179868 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:40.179947 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:40.179923 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:40.180014 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:40.180000 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:40.378848 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:40.378808 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:41.358526 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:41.358492 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:41.507065 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:41.507031 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:42.179709 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:42.179670 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:42.179709 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:42.179705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:42.179954 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:42.179787 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:42.179954 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:42.179805 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:42.179954 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:42.179906 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:42.180147 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:42.180018 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:44.180919 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:44.180884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:44.180919 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:44.180918 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:44.181397 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:44.181001 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:44.181397 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:44.181066 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:44.181397 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:44.181092 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:44.181397 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:44.181154 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:44.774966 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:44.774924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:44.959707 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:44.959674 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:45.153363 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:45.153334 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:45.298719 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:45.298683 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:45.444820 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:45.444742 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:45.645127 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:45.645085 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:46.180536 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:46.180329 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:46.180684 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:46.180338 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:46.180684 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:46.180579 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:46.180797 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:46.180674 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:46.180797 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:46.180377 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:46.180900 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:46.180792 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:46.361747 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:46.361717 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-69x5r"] Nov 02 14:57:46.365368 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:46.365338 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6p7c"] Nov 02 14:57:46.367004 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:46.366968 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sx5r5"] Nov 02 14:57:46.409777 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:46.409754 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:46.409777 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:46.409766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:46.409958 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:46.409754 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:46.409958 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:46.409852 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:46.409958 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:46.409913 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:46.410103 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:46.409986 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:47.777509 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:47.777481 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:47.923381 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:47.923347 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:48.077653 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:48.077569 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:48.179910 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:48.179877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:48.179910 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:48.179892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:48.180258 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:48.179877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:48.180258 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:48.180012 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:48.180258 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:48.180078 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:48.180258 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:48.180167 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:48.847241 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:48.847204 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:49.148897 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:49.148862 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:50.182656 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:50.182625 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:50.183197 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:50.182743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:50.183197 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:50.182746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:50.183197 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:50.182743 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sx5r5" podUID="6c7d4401-1b82-4665-8838-9e05fac70e69" Nov 02 14:57:50.183197 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:50.182820 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6p7c" podUID="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" Nov 02 14:57:50.183197 ip-10-0-135-181 kubenswrapper[2577]: E1102 14:57:50.182911 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69x5r" podUID="ed84914e-49d4-4f14-877b-ca1b0cdff144" Nov 02 14:57:50.928573 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:50.928541 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeReady" Nov 02 14:57:50.928739 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:50.928674 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 14:57:51.076597 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.076517 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4njs7"] Nov 02 14:57:51.078728 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.078713 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vr5nl"] Nov 02 14:57:51.078854 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.078834 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.080451 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.080432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:57:51.122804 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.122771 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:51.133477 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.133444 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vr5nl"] Nov 02 14:57:51.135533 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.135503 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 14:57:51.137865 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.137832 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4njs7"] Nov 02 14:57:51.138021 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.137949 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lzs9f"] Nov 02 14:57:51.140936 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.140915 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.143476 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.143458 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 14:57:51.143746 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.143731 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4d7td\"" Nov 02 14:57:51.143839 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.143821 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 14:57:51.143888 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.143825 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 14:57:51.143947 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.143928 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c2bv5\"" Nov 02 14:57:51.144002 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.143946 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 14:57:51.144002 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.143936 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 14:57:51.192365 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.192332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-data-volume\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.192812 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.192394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-crio-socket\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.192812 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.192484 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-cert\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:57:51.192812 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.192577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.192812 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.192599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.192812 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.192702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtd5\" (UniqueName: \"kubernetes.io/projected/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-api-access-7jtd5\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.192812 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.192726 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ll25\" (UniqueName: \"kubernetes.io/projected/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-kube-api-access-7ll25\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:57:51.196553 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.196530 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lzs9f"] Nov 02 14:57:51.235285 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.235259 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 14:57:51.235475 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.235458 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cr8rg\"" Nov 02 14:57:51.236108 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.236092 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 14:57:51.236165 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.236097 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 14:57:51.272613 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.272580 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:51.293798 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.293767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.293929 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.293807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.293929 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.293860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtd5\" (UniqueName: \"kubernetes.io/projected/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-api-access-7jtd5\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.293929 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.293882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ll25\" (UniqueName: \"kubernetes.io/projected/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-kube-api-access-7ll25\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:57:51.293929 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.293910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-data-volume\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.294167 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.293938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-metrics-tls\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.294167 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.294016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-crio-socket\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.294167 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.294044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-config-volume\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.294167 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.294076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-cert\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:57:51.294167 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.294115 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-929p7\" (UniqueName: \"kubernetes.io/projected/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-kube-api-access-929p7\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.294400 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.294197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-tmp-dir\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.294400 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.294269 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-crio-socket\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.294400 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.294317 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-data-volume\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.294633 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.294607 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.297651 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.297630 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.297754 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.297652 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-cert\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:57:51.359253 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.359183 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtd5\" (UniqueName: \"kubernetes.io/projected/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-api-access-7jtd5\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.359828 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.359805 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ll25\" (UniqueName: \"kubernetes.io/projected/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-kube-api-access-7ll25\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:57:51.393412 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.393382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:57:51.394582 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.394561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-metrics-tls\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.394654 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.394610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-config-volume\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.394654 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.394635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-929p7\" (UniqueName: \"kubernetes.io/projected/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-kube-api-access-929p7\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.394654 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.394651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-tmp-dir\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.394936 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.394919 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-tmp-dir\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.395149 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.395130 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-config-volume\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.395814 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.395797 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:57:51.396952 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.396934 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-metrics-tls\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.456887 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.456815 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-929p7\" (UniqueName: \"kubernetes.io/projected/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-kube-api-access-929p7\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.752209 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.752158 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:51.825870 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.825847 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vr5nl"] Nov 02 14:57:51.827561 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:51.827538 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7df6cc_81c7_4ecc_b1f8_4b8de2464445.slice/crio-539399b817160e8f3020c6fc70c3ce2c558700e8f63f51be00e40c9ebcaab21a WatchSource:0}: Error finding container 539399b817160e8f3020c6fc70c3ce2c558700e8f63f51be00e40c9ebcaab21a: Status 404 returned error can't find the container with id 539399b817160e8f3020c6fc70c3ce2c558700e8f63f51be00e40c9ebcaab21a Nov 02 14:57:51.828404 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.828380 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4njs7"] Nov 02 14:57:51.985965 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:51.985929 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:52.116080 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:57:52.116047 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f454c8_54e1_4eba_a8ea_10a32729ce0e.slice/crio-109c1a32d0a0161252216f1d08997d3ba0d57b1706f1104181079d0c9ee20329 WatchSource:0}: Error finding container 109c1a32d0a0161252216f1d08997d3ba0d57b1706f1104181079d0c9ee20329: Status 404 returned error can't find the container with id 109c1a32d0a0161252216f1d08997d3ba0d57b1706f1104181079d0c9ee20329 Nov 02 14:57:52.117743 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.117720 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lzs9f"] Nov 02 14:57:52.179964 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.179927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:57:52.180130 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.179927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:57:52.180130 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.179933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:57:52.244290 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.244251 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:57:52.244290 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.244265 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:57:52.245531 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.245514 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:57:52.245793 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.245777 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-29n45\"" Nov 02 14:57:52.245793 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.245789 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gmvcm\"" Nov 02 14:57:52.245932 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.245800 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:57:52.245932 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.245813 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gwmlb\"" Nov 02 14:57:52.425128 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.425099 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lzs9f" event={"ID":"d4f454c8-54e1-4eba-a8ea-10a32729ce0e","Type":"ContainerStarted","Data":"109c1a32d0a0161252216f1d08997d3ba0d57b1706f1104181079d0c9ee20329"} Nov 02 14:57:52.426206 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.426183 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vr5nl" event={"ID":"888654bd-3ef7-4ff1-b31e-0ce4be3587b6","Type":"ContainerStarted","Data":"2efad5608fa771bcbbb1e1c0d5f8ffc5ff8e8ba88284fc22b365b15408d69338"} Nov 02 14:57:52.427613 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.427587 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4njs7" event={"ID":"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445","Type":"ContainerStarted","Data":"cc7d0c4ecba74de261e90b28579a599fb31fcf276ad9343e979bc16c1999fc31"} Nov 02 14:57:52.427703 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:52.427632 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4njs7" event={"ID":"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445","Type":"ContainerStarted","Data":"539399b817160e8f3020c6fc70c3ce2c558700e8f63f51be00e40c9ebcaab21a"} Nov 02 14:57:53.149963 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:53.149932 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:53.335670 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:53.335630 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:53.432755 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:53.432721 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4njs7" event={"ID":"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445","Type":"ContainerStarted","Data":"93105d7a15f8cd40dae443ee43df45648b33a57ca1993e41d0203a3e51275e0c"} Nov 02 14:57:53.518691 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:53.518655 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:53.948752 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:53.948705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:55.440914 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:55.440804 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lzs9f" event={"ID":"d4f454c8-54e1-4eba-a8ea-10a32729ce0e","Type":"ContainerStarted","Data":"d666d627a02ddf0f628febf11e3b8a4bcaf02ec5768d4b187df75076ce4f267f"} Nov 02 14:57:55.440914 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:55.440895 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lzs9f" event={"ID":"d4f454c8-54e1-4eba-a8ea-10a32729ce0e","Type":"ContainerStarted","Data":"8f7636dd5fa98cb118d0cbe62c9d060b5a907d2725dee2a68695bdf0e7a1ebd0"} Nov 02 14:57:55.441337 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:55.440989 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lzs9f" Nov 02 14:57:55.442781 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:55.442751 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vr5nl" event={"ID":"888654bd-3ef7-4ff1-b31e-0ce4be3587b6","Type":"ContainerStarted","Data":"05aa86963a4303cf89878240443e75a68af5129b2739fb3f5a33cddc077ed444"} Nov 02 14:57:55.563578 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:55.563530 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:55.564204 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:55.564164 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-lzs9f" podStartSLOduration=2.144028132 podStartE2EDuration="4.56415203s" totalImagesPullingTime="2.420123898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:55.563696035 +0000 UTC m=+51.925522002" watchObservedRunningTime="2026-11-02 14:57:55.56415203 +0000 UTC m=+51.925977988" Nov 02 14:57:55.683102 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:55.683038 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vr5nl" podStartSLOduration=2.353392214 podStartE2EDuration="5.683017558s" totalImagesPullingTime="3.329625344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:55.682466784 +0000 UTC m=+52.044292752" watchObservedRunningTime="2026-11-02 14:57:55.683017558 +0000 UTC m=+52.044843526" Nov 02 14:57:55.718131 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:55.718044 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:56.447496 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:56.447457 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4njs7" event={"ID":"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445","Type":"ContainerStarted","Data":"17585ed0556cfe7c5a4114ff0333d6e8c22f3e28aaa650a889f3cb1e01b375df"} Nov 02 14:57:56.570104 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:56.570050 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4njs7" podStartSLOduration=2.794345529 podStartE2EDuration="6.570035625s" totalImagesPullingTime="3.775690096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:56.568483186 +0000 UTC m=+52.930309153" watchObservedRunningTime="2026-11-02 14:57:56.570035625 +0000 UTC m=+52.931861595" Nov 02 14:57:58.990913 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:58.990877 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:59.476189 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:59.476149 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:59.627612 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:59.627574 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:57:59.998233 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:57:59.998199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:00.200533 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.200497 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:00.209629 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.209601 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q"] Nov 02 14:58:00.211733 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.211718 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.273655 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.273567 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d4l7z"] Nov 02 14:58:00.276113 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.276095 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.323224 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.323196 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 14:58:00.323224 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.323197 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 14:58:00.323517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.323496 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 14:58:00.323561 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.323539 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 14:58:00.327319 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.327301 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2bsvz\"" Nov 02 14:58:00.359699 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.359665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.359808 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.359740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.359808 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.359791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96gzz\" (UniqueName: \"kubernetes.io/projected/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-kube-api-access-96gzz\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.359906 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.359889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.374358 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.374332 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 14:58:00.374462 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.374383 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 14:58:00.375175 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.375154 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q"] Nov 02 14:58:00.375349 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.375335 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 14:58:00.375679 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.375664 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zw4g4\"" Nov 02 14:58:00.375679 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.375670 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 14:58:00.381762 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.381742 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk"] Nov 02 14:58:00.384101 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.384088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.422414 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.422385 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:00.453166 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.453134 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk"] Nov 02 14:58:00.460539 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-tls\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.460662 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.460662 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-wtmp\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.460662 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-metrics-client-ca\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.460805 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.460805 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7dn7\" (UniqueName: \"kubernetes.io/projected/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-kube-api-access-w7dn7\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.460897 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-root\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.460897 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-sys\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.461018 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-textfile\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.461018 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.461018 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.460938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gzz\" (UniqueName: \"kubernetes.io/projected/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-kube-api-access-96gzz\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.461173 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.461129 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.461173 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.461152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.461451 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.461432 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.463323 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.463306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.463459 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.463437 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.472373 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.472350 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 14:58:00.474999 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.474965 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bxvmm\"" Nov 02 14:58:00.475049 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.475001 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 14:58:00.475129 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.475117 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 14:58:00.524554 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.524497 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96gzz\" (UniqueName: \"kubernetes.io/projected/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-kube-api-access-96gzz\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.562082 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-textfile\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562082 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7m4\" (UniqueName: \"kubernetes.io/projected/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-api-access-bn7m4\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.562326 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd0ae468-b91d-4121-baca-0cc3836997ae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.562326 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.562326 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562326 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-wtmp\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-metrics-client-ca\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.562517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562451 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-textfile\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562453 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.562517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562468 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-wtmp\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562755 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-sys\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562755 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-tls\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562755 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562645 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-sys\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562755 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.562755 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.562755 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dn7\" (UniqueName: \"kubernetes.io/projected/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-kube-api-access-w7dn7\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.563053 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-root\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.563053 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562825 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-metrics-client-ca\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.563053 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.562841 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-root\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.563206 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.563183 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.564733 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.564709 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.564833 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.564819 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-tls\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.599705 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.599672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:00.625260 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.625231 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dn7\" (UniqueName: \"kubernetes.io/projected/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-kube-api-access-w7dn7\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.663784 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.663748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.663941 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.663806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7m4\" (UniqueName: \"kubernetes.io/projected/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-api-access-bn7m4\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.663941 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.663826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd0ae468-b91d-4121-baca-0cc3836997ae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.663941 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.663854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.663941 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.663875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.663941 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.663893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.664364 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.664341 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd0ae468-b91d-4121-baca-0cc3836997ae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.664639 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.664614 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.664713 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.664649 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.666364 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.666344 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.666435 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.666421 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.726040 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.726003 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7m4\" (UniqueName: \"kubernetes.io/projected/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-api-access-bn7m4\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:00.759672 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.759647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:00.824192 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.824075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:00.888544 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.888480 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:00.897707 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:58:00.897682 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7b659d_dcaa_40e1_ac29_5cdc7a619a4f.slice/crio-2fc621fd8ae7d5e63ee52805d29176009c9c425652cc3f205650c69783e5f9e8 WatchSource:0}: Error finding container 2fc621fd8ae7d5e63ee52805d29176009c9c425652cc3f205650c69783e5f9e8: Status 404 returned error can't find the container with id 2fc621fd8ae7d5e63ee52805d29176009c9c425652cc3f205650c69783e5f9e8 Nov 02 14:58:00.996781 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:00.996740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:01.136968 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:01.136935 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:01.242509 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:58:01.242477 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b3ae42_1fb7_4068_80bd_a8c94e1d35f3.slice/crio-b2e1eda306ef00dba83f3cfbc19161636503b8132a5c04c5f2c5390235130f1d WatchSource:0}: Error finding container b2e1eda306ef00dba83f3cfbc19161636503b8132a5c04c5f2c5390235130f1d: Status 404 returned error can't find the container with id b2e1eda306ef00dba83f3cfbc19161636503b8132a5c04c5f2c5390235130f1d Nov 02 14:58:01.243618 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:01.243587 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q"] Nov 02 14:58:01.369520 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:58:01.369493 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0ae468_b91d_4121_baca_0cc3836997ae.slice/crio-9617a9080a2344017c32fbbe264d351f4023d7f09c96491cb5cfa2687752714a WatchSource:0}: Error finding container 9617a9080a2344017c32fbbe264d351f4023d7f09c96491cb5cfa2687752714a: Status 404 returned error can't find the container with id 9617a9080a2344017c32fbbe264d351f4023d7f09c96491cb5cfa2687752714a Nov 02 14:58:01.371310 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:01.371277 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk"] Nov 02 14:58:01.462868 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:01.462793 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" event={"ID":"bd0ae468-b91d-4121-baca-0cc3836997ae","Type":"ContainerStarted","Data":"9617a9080a2344017c32fbbe264d351f4023d7f09c96491cb5cfa2687752714a"} Nov 02 14:58:01.464092 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:01.464063 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4l7z" event={"ID":"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f","Type":"ContainerStarted","Data":"2fc621fd8ae7d5e63ee52805d29176009c9c425652cc3f205650c69783e5f9e8"} Nov 02 14:58:01.465818 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:01.465798 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" event={"ID":"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3","Type":"ContainerStarted","Data":"ae1b6140e5dfef7cdf50392c9863294cdc9563d7273efde522df24f16fcaceeb"} Nov 02 14:58:01.465901 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:01.465836 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" event={"ID":"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3","Type":"ContainerStarted","Data":"7bcad862b63c05b2abcf333aa7368e73c5f81df7f86887e470268518e98022d1"} Nov 02 14:58:01.465901 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:01.465855 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" event={"ID":"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3","Type":"ContainerStarted","Data":"b2e1eda306ef00dba83f3cfbc19161636503b8132a5c04c5f2c5390235130f1d"} Nov 02 14:58:01.643027 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:01.642964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:02.030351 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:02.030317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:02.552721 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:02.552693 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:02.837532 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:02.837491 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:03.007441 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.007352 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:03.184898 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.184865 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b"] Nov 02 14:58:03.187616 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.187593 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.194873 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.194844 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:03.203005 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.201239 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b"] Nov 02 14:58:03.244805 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.244778 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 14:58:03.244951 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.244937 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 14:58:03.247547 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.247530 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:03.250704 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.250683 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 14:58:03.250845 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.250806 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-67fcni9nmp7r6\"" Nov 02 14:58:03.250845 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.250829 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 14:58:03.250960 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.250849 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dcs5n\"" Nov 02 14:58:03.251021 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.250959 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 14:58:03.289150 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.289122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jcrc\" (UniqueName: \"kubernetes.io/projected/83a2eda6-8157-428a-b0f3-c2cc77563252-kube-api-access-5jcrc\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.289334 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.289155 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-grpc-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.289334 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.289185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.289334 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.289303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.289575 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.289367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.289575 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.289448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83a2eda6-8157-428a-b0f3-c2cc77563252-metrics-client-ca\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.289575 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.289475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a2eda6-8157-428a-b0f3-c2cc77563252-tmp\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.289575 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.289551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.289751 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.289577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.390102 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.390209 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.390209 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcrc\" (UniqueName: \"kubernetes.io/projected/83a2eda6-8157-428a-b0f3-c2cc77563252-kube-api-access-5jcrc\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.390330 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390200 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:03.390330 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-grpc-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.390330 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.391194 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.391194 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.391194 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83a2eda6-8157-428a-b0f3-c2cc77563252-metrics-client-ca\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.391194 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.390849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a2eda6-8157-428a-b0f3-c2cc77563252-tmp\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.391783 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.391756 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83a2eda6-8157-428a-b0f3-c2cc77563252-metrics-client-ca\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.392034 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.392009 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a2eda6-8157-428a-b0f3-c2cc77563252-tmp\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.393309 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.393277 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.393407 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.393313 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.393524 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.393502 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.393824 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.393805 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-grpc-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.393890 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.393810 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.395061 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.395043 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.460870 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.460833 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcrc\" (UniqueName: \"kubernetes.io/projected/83a2eda6-8157-428a-b0f3-c2cc77563252-kube-api-access-5jcrc\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.474653 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.474625 2577 generic.go:356] "Generic (PLEG): container finished" podID="ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f" containerID="e3caca96c848414dd05b1f3ef525325c93c3b33b17dfbf27fe77d1e0cb01358f" exitCode=0 Nov 02 14:58:03.474816 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.474705 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4l7z" event={"ID":"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f","Type":"ContainerDied","Data":"e3caca96c848414dd05b1f3ef525325c93c3b33b17dfbf27fe77d1e0cb01358f"} Nov 02 14:58:03.479345 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.479316 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" event={"ID":"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3","Type":"ContainerStarted","Data":"4b310c69d377552d6160162681cd94f9cf79b40bb3d5cc90722831b26c5e1b90"} Nov 02 14:58:03.481558 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.481537 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" event={"ID":"bd0ae468-b91d-4121-baca-0cc3836997ae","Type":"ContainerStarted","Data":"8db32bb2b0a2b79eaf18a43738508c4bae65a2d9c983fdedcdd2d22b671122df"} Nov 02 14:58:03.481624 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.481587 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" event={"ID":"bd0ae468-b91d-4121-baca-0cc3836997ae","Type":"ContainerStarted","Data":"c805a8500f173a7a002b77b3badec04b9c5dfb253a10cfaa3d07b1a225444869"} Nov 02 14:58:03.481624 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.481618 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" event={"ID":"bd0ae468-b91d-4121-baca-0cc3836997ae","Type":"ContainerStarted","Data":"26b16fc3bcc4c1ccbae23fae7bb005f5ae9fd1353e40d310f96c22c27613b316"} Nov 02 14:58:03.512800 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.512778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:03.580001 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.579864 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:03.681461 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.681406 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" podStartSLOduration=1.713787835 podStartE2EDuration="3.681392208s" totalImagesPullingTime="1.967604373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:03.681201141 +0000 UTC m=+60.043027102" watchObservedRunningTime="2026-11-02 14:58:03.681392208 +0000 UTC m=+60.043218174" Nov 02 14:58:03.934795 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.934732 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" podStartSLOduration=2.18367321 podStartE2EDuration="3.934715253s" totalImagesPullingTime="1.751042043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:03.933666323 +0000 UTC m=+60.295492291" watchObservedRunningTime="2026-11-02 14:58:03.934715253 +0000 UTC m=+60.296541222" Nov 02 14:58:03.938058 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:58:03.938030 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a2eda6_8157_428a_b0f3_c2cc77563252.slice/crio-c6ad51e734f9ef12ffa6eb9eef9a56b2b3e4aa191e810365072c35563dbdda99 WatchSource:0}: Error finding container c6ad51e734f9ef12ffa6eb9eef9a56b2b3e4aa191e810365072c35563dbdda99: Status 404 returned error can't find the container with id c6ad51e734f9ef12ffa6eb9eef9a56b2b3e4aa191e810365072c35563dbdda99 Nov 02 14:58:03.940386 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:03.940189 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b"] Nov 02 14:58:04.356104 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.356031 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:04.486204 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.486169 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" event={"ID":"83a2eda6-8157-428a-b0f3-c2cc77563252","Type":"ContainerStarted","Data":"c6ad51e734f9ef12ffa6eb9eef9a56b2b3e4aa191e810365072c35563dbdda99"} Nov 02 14:58:04.488349 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.488320 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4l7z" event={"ID":"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f","Type":"ContainerStarted","Data":"d33769da0ed36d5bf9559ba4aebbe36f7180c223444ba7d54f5b9b3091f6c663"} Nov 02 14:58:04.488510 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.488375 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4l7z" event={"ID":"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f","Type":"ContainerStarted","Data":"19a8be723c487121235647cfdced81e1786e8698782d51306f8888e69a273374"} Nov 02 14:58:04.525547 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.525525 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:04.610535 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.610457 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d4l7z" podStartSLOduration=2.878402961 podStartE2EDuration="4.610444663s" totalImagesPullingTime="1.647266694s" totalInitContainerRuntime="84.775008ms" isStatefulPod=true podCreationTimestamp="2026-11-02 14:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:04.609881843 +0000 UTC m=+60.971707804" watchObservedRunningTime="2026-11-02 14:58:04.610444663 +0000 UTC m=+60.972270631" Nov 02 14:58:04.671254 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.671229 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:04.874129 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.874103 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:04.916708 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.916675 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b"] Nov 02 14:58:04.918877 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.918861 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:04.974070 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.974039 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 14:58:04.985006 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:04.984084 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b"] Nov 02 14:58:05.003861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.003837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phl58\" (UniqueName: \"kubernetes.io/projected/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-kube-api-access-phl58\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:05.004008 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.003901 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:05.021447 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.021416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:05.075953 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.075923 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-djmct\"" Nov 02 14:58:05.104761 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.104733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:05.104904 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.104829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phl58\" (UniqueName: \"kubernetes.io/projected/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-kube-api-access-phl58\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:05.107889 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.107853 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:05.170554 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.170488 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phl58\" (UniqueName: \"kubernetes.io/projected/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-kube-api-access-phl58\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:05.230551 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.230517 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:05.335339 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.335309 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-74bdbddd8-wl9lp"] Nov 02 14:58:05.338650 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.338630 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.396509 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.395689 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 14:58:05.396509 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.396091 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 14:58:05.396509 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.396118 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fgvhh\"" Nov 02 14:58:05.396509 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.396226 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 14:58:05.396509 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.396304 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2bv74s1v9qjn0\"" Nov 02 14:58:05.396509 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.396424 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 14:58:05.397109 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.397086 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74bdbddd8-wl9lp"] Nov 02 14:58:05.406917 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.406887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxbs\" (UniqueName: \"kubernetes.io/projected/e30a530f-5e3c-48f7-8872-f0b58d08a91e-kube-api-access-2hxbs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.407058 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.406945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e30a530f-5e3c-48f7-8872-f0b58d08a91e-audit-log\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.407170 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.407154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-tls\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.407263 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.407187 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-client-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.407263 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.407210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-metrics-server-audit-profiles\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.407381 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.407357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.407431 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.407401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-client-certs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.449579 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.449506 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:05.508093 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.508059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-tls\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.508263 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.508104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-client-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.508562 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.508532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-metrics-server-audit-profiles\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.508674 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.508608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.508674 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.508648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-client-certs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.508770 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.508714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxbs\" (UniqueName: \"kubernetes.io/projected/e30a530f-5e3c-48f7-8872-f0b58d08a91e-kube-api-access-2hxbs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.508819 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.508769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e30a530f-5e3c-48f7-8872-f0b58d08a91e-audit-log\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.509773 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.509580 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e30a530f-5e3c-48f7-8872-f0b58d08a91e-audit-log\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.509773 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.509731 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.509773 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.509587 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-metrics-server-audit-profiles\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.511203 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.511178 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-tls\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.512328 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.512309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-client-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.512612 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.512590 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-client-certs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.572356 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.572316 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxbs\" (UniqueName: \"kubernetes.io/projected/e30a530f-5e3c-48f7-8872-f0b58d08a91e-kube-api-access-2hxbs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:05.601381 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:58:05.601341 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3d2fcf_47c8_4c96_b493_4551ca3a9bf6.slice/crio-cdcdb3a8d3ecfa711d4d7abc02ea3a685040982f19b5170110b6a82035e5c337 WatchSource:0}: Error finding container cdcdb3a8d3ecfa711d4d7abc02ea3a685040982f19b5170110b6a82035e5c337: Status 404 returned error can't find the container with id cdcdb3a8d3ecfa711d4d7abc02ea3a685040982f19b5170110b6a82035e5c337 Nov 02 14:58:05.602403 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.602376 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b"] Nov 02 14:58:05.651648 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:05.651620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:06.070831 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:58:06.070799 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30a530f_5e3c_48f7_8872_f0b58d08a91e.slice/crio-cdd0264f39a453d8b6ed93bd8362b977e34fd8d43d77989f7a7ca45abe9e0bfc WatchSource:0}: Error finding container cdd0264f39a453d8b6ed93bd8362b977e34fd8d43d77989f7a7ca45abe9e0bfc: Status 404 returned error can't find the container with id cdd0264f39a453d8b6ed93bd8362b977e34fd8d43d77989f7a7ca45abe9e0bfc Nov 02 14:58:06.071086 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:06.071058 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74bdbddd8-wl9lp"] Nov 02 14:58:06.498678 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:06.498635 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" event={"ID":"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6","Type":"ContainerStarted","Data":"cdcdb3a8d3ecfa711d4d7abc02ea3a685040982f19b5170110b6a82035e5c337"} Nov 02 14:58:06.499989 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:06.499953 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" event={"ID":"e30a530f-5e3c-48f7-8872-f0b58d08a91e","Type":"ContainerStarted","Data":"cdd0264f39a453d8b6ed93bd8362b977e34fd8d43d77989f7a7ca45abe9e0bfc"} Nov 02 14:58:07.488138 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:07.488107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:07.507552 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:07.507526 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" event={"ID":"83a2eda6-8157-428a-b0f3-c2cc77563252","Type":"ContainerStarted","Data":"d6d590a7d50b70a3c97d67ae205001c07ca17138b6ed9368aa9bcb42d6138717"} Nov 02 14:58:07.507660 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:07.507599 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" event={"ID":"83a2eda6-8157-428a-b0f3-c2cc77563252","Type":"ContainerStarted","Data":"ed651b772425ffdab66fceb1b8ef56d72713bce0b14e9d2f5a8bcb817868658e"} Nov 02 14:58:07.507660 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:07.507624 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" event={"ID":"83a2eda6-8157-428a-b0f3-c2cc77563252","Type":"ContainerStarted","Data":"cb39dd3f16afe6759a0811250d3c5e1fd2d659a4f982ca69ee19496c41dac7e7"} Nov 02 14:58:07.633989 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:07.633945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:07.811783 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:07.811706 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:07.957807 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:07.957773 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:08.512722 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.512686 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" event={"ID":"e30a530f-5e3c-48f7-8872-f0b58d08a91e","Type":"ContainerStarted","Data":"54aebad7ed18632ecf543f9fe28912db8b042e4c036baa372015b7a9c34064a5"} Nov 02 14:58:08.514362 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.514333 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" event={"ID":"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6","Type":"ContainerStarted","Data":"c90fcf357257ec19333adc60176661aaa7c1261bd9e2f7a7a660c6eb0897afac"} Nov 02 14:58:08.514516 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.514488 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:08.517444 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.517422 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" event={"ID":"83a2eda6-8157-428a-b0f3-c2cc77563252","Type":"ContainerStarted","Data":"6ed0e8dddc454d70cdbde5bf8eed378ab2d80be72e598ceacebef10dce762a66"} Nov 02 14:58:08.517562 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.517473 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" event={"ID":"83a2eda6-8157-428a-b0f3-c2cc77563252","Type":"ContainerStarted","Data":"f798a5a96b7363a10f6cba605b6b525791fdc1eed54e59144b003c4468227d09"} Nov 02 14:58:08.517562 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.517503 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" event={"ID":"83a2eda6-8157-428a-b0f3-c2cc77563252","Type":"ContainerStarted","Data":"e8e7cf783b43c2d36d07abf87cc26df6fa8b66b141a0e3775bd0a031cb8caeb6"} Nov 02 14:58:08.517562 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.517534 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:08.519837 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.519815 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:08.645543 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.645491 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" podStartSLOduration=1.600688439 podStartE2EDuration="3.645477099s" totalImagesPullingTime="2.04478866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:08.644772644 +0000 UTC m=+65.006598612" watchObservedRunningTime="2026-11-02 14:58:08.645477099 +0000 UTC m=+65.007303063" Nov 02 14:58:08.764633 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.764531 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" podStartSLOduration=2.884744176 podStartE2EDuration="4.764515213s" totalImagesPullingTime="1.879771037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:08.76394015 +0000 UTC m=+65.125766118" watchObservedRunningTime="2026-11-02 14:58:08.764515213 +0000 UTC m=+65.126341179" Nov 02 14:58:08.891467 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:08.891412 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" podStartSLOduration=2.084364618 podStartE2EDuration="5.891399877s" totalImagesPullingTime="3.807035259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:08.890032218 +0000 UTC m=+65.251858184" watchObservedRunningTime="2026-11-02 14:58:08.891399877 +0000 UTC m=+65.253225844" Nov 02 14:58:09.741764 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:09.741736 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:10.055563 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.055472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:58:10.055726 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.055644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:10.109158 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.109117 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:58:10.118452 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.118425 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:58:10.156032 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.156000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:58:10.159748 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.159725 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:58:10.168909 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.168874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:10.211828 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.211795 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:58:10.250505 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.250480 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gmvcm\"" Nov 02 14:58:10.253565 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.253554 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gwmlb\"" Nov 02 14:58:10.258796 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.258779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:58:10.261254 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.261232 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:10.269259 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.269240 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:58:10.279965 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.279935 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:58:10.543861 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.543835 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-29n45\"" Nov 02 14:58:10.551891 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.551872 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:58:10.683226 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:58:10.683198 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded84914e_49d4_4f14_877b_ca1b0cdff144.slice/crio-07acbdd0495d2ebd0aeac4de1d2fb585afafe5f17bd5420c7eb7cfced00df472 WatchSource:0}: Error finding container 07acbdd0495d2ebd0aeac4de1d2fb585afafe5f17bd5420c7eb7cfced00df472: Status 404 returned error can't find the container with id 07acbdd0495d2ebd0aeac4de1d2fb585afafe5f17bd5420c7eb7cfced00df472 Nov 02 14:58:10.685938 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.685919 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-69x5r"] Nov 02 14:58:10.688801 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.688783 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sx5r5"] Nov 02 14:58:10.720429 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:58:10.720403 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7d4401_1b82_4665_8838_9e05fac70e69.slice/crio-c10c3fc3c2de2e1822ec788cb9e97f58911c8a5559f1b73b0ffcdedf49328414 WatchSource:0}: Error finding container c10c3fc3c2de2e1822ec788cb9e97f58911c8a5559f1b73b0ffcdedf49328414: Status 404 returned error can't find the container with id c10c3fc3c2de2e1822ec788cb9e97f58911c8a5559f1b73b0ffcdedf49328414 Nov 02 14:58:10.924389 ip-10-0-135-181 kubenswrapper[2577]: W1102 14:58:10.924355 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc0d2d5_f127_4b55_bed1_0bbfffb31ff4.slice/crio-9e63a570836e0675f700db3e145705fc4b1a2d5f2af7361e8081175f513adb10 WatchSource:0}: Error finding container 9e63a570836e0675f700db3e145705fc4b1a2d5f2af7361e8081175f513adb10: Status 404 returned error can't find the container with id 9e63a570836e0675f700db3e145705fc4b1a2d5f2af7361e8081175f513adb10 Nov 02 14:58:10.926394 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:10.926371 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6p7c"] Nov 02 14:58:11.536010 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:11.535960 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-69x5r" event={"ID":"ed84914e-49d4-4f14-877b-ca1b0cdff144","Type":"ContainerStarted","Data":"07acbdd0495d2ebd0aeac4de1d2fb585afafe5f17bd5420c7eb7cfced00df472"} Nov 02 14:58:11.537307 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:11.537279 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sx5r5" event={"ID":"6c7d4401-1b82-4665-8838-9e05fac70e69","Type":"ContainerStarted","Data":"c10c3fc3c2de2e1822ec788cb9e97f58911c8a5559f1b73b0ffcdedf49328414"} Nov 02 14:58:11.538430 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:11.538405 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6p7c" event={"ID":"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4","Type":"ContainerStarted","Data":"9e63a570836e0675f700db3e145705fc4b1a2d5f2af7361e8081175f513adb10"} Nov 02 14:58:12.233708 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:12.233673 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:12.386487 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:12.386437 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:12.544308 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:12.544272 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sx5r5" event={"ID":"6c7d4401-1b82-4665-8838-9e05fac70e69","Type":"ContainerStarted","Data":"1083515388c0f3096f55d71ef87494bed37072119ee642e2520fdca69271d33d"} Nov 02 14:58:13.552386 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:13.552348 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sx5r5" event={"ID":"6c7d4401-1b82-4665-8838-9e05fac70e69","Type":"ContainerStarted","Data":"e8e1da8a1d1a139aaae384c0424012230d5ed32220b2b0b25da4c50646ca868b"} Nov 02 14:58:13.587166 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:13.587137 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:13.676439 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:13.676376 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sx5r5" podStartSLOduration=68.002628646 podStartE2EDuration="1m9.676355096s" totalImagesPullingTime="1.67372645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:13.675186495 +0000 UTC m=+70.037012466" watchObservedRunningTime="2026-11-02 14:58:13.676355096 +0000 UTC m=+70.038181065" Nov 02 14:58:13.795517 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:13.795479 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:13.993580 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:13.993537 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:14.206093 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:14.205794 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:14.355777 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:14.355704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:14.528514 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:14.528481 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:15.562872 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:15.562831 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6p7c" event={"ID":"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4","Type":"ContainerStarted","Data":"aa149640a5e39a6f0874700977ff604c264565093538b4b12ba18fb1d51fcd5a"} Nov 02 14:58:15.563389 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:15.563028 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:58:15.689181 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:15.689115 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l6p7c" podStartSLOduration=67.778969071 podStartE2EDuration="1m11.689098847s" totalImagesPullingTime="3.910129776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:15.687186412 +0000 UTC m=+72.049012380" watchObservedRunningTime="2026-11-02 14:58:15.689098847 +0000 UTC m=+72.050924816" Nov 02 14:58:15.771191 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:15.771145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:15.952885 ip-10-0-135-181 kubenswrapper[2577]: I1102 14:58:15.952851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:16.263015 ip-10-0-135-181 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 14:58:16.280402 ip-10-0-135-181 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 14:58:16.280570 ip-10-0-135-181 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 14:58:16.280622 ip-10-0-135-181 systemd[1]: kubelet.service: Consumed 4.151s CPU time. Nov 02 14:58:16.295655 ip-10-0-135-181 systemd[1]: Starting Kubernetes Kubelet... Nov 02 14:58:16.358739 ip-10-0-135-181 kubenswrapper[5433]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:58:16.358739 ip-10-0-135-181 kubenswrapper[5433]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 14:58:16.358739 ip-10-0-135-181 kubenswrapper[5433]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:58:16.358739 ip-10-0-135-181 kubenswrapper[5433]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:58:16.360856 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360773 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:16.360856 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360855 5433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:16.360856 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360859 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360863 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360866 5433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360869 5433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360872 5433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360874 5433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360877 5433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360880 5433 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360882 5433 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360885 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360888 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360891 5433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360894 5433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360897 5433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360899 5433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360902 5433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360906 5433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360911 5433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360914 5433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360917 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:16.360964 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360920 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360923 5433 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360926 5433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360928 5433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360932 5433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360934 5433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360937 5433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360939 5433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360942 5433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360945 5433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360947 5433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360950 5433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360953 5433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360957 5433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360960 5433 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360963 5433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360966 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360968 5433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360984 5433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360991 5433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:16.361485 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360994 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.360997 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361000 5433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361003 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361006 5433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361009 5433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361013 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361016 5433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361018 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361021 5433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361023 5433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361026 5433 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361028 5433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361031 5433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361035 5433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361039 5433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361042 5433 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361045 5433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361048 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:16.361965 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361051 5433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361054 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361057 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361059 5433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361063 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361066 5433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361068 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361071 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361074 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361076 5433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361079 5433 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361081 5433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361084 5433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361087 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361090 5433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361092 5433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361095 5433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361097 5433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361106 5433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:16.362483 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361109 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361111 5433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361114 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361116 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361118 5433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361121 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361124 5433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361127 5433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361129 5433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361132 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361134 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361137 5433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361139 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361142 5433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361144 5433 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361148 5433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361151 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361155 5433 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361157 5433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361160 5433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:16.362943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361162 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361165 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361167 5433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361170 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361172 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.361175 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362174 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362182 5433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362186 5433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362189 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362192 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362195 5433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362204 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362207 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362209 5433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362212 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362214 5433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362217 5433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362220 5433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:16.363551 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362222 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362225 5433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362227 5433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362231 5433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362235 5433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362238 5433 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362241 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362243 5433 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362246 5433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362249 5433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362252 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362255 5433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362258 5433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362260 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362263 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362266 5433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362268 5433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362271 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362273 5433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362276 5433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:16.364033 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362279 5433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362281 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362283 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362286 5433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362289 5433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362291 5433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362300 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362302 5433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362305 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362308 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362310 5433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362313 5433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362315 5433 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362318 5433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362321 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362324 5433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362326 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362329 5433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362332 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:16.364521 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362335 5433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362337 5433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362340 5433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362343 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362346 5433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362348 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362351 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362354 5433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362358 5433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362362 5433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362365 5433 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362368 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362371 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362374 5433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362376 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362379 5433 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362381 5433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362384 5433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362387 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362390 5433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:16.365041 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362398 5433 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362401 5433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362404 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362406 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362409 5433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362411 5433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362414 5433 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362416 5433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362419 5433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362422 5433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362424 5433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362427 5433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362429 5433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362432 5433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362435 5433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362438 5433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362441 5433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362444 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362446 5433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362449 5433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:16.365527 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362451 5433 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362454 5433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362457 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362459 5433 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362462 5433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362464 5433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362467 5433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362470 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362472 5433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362475 5433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362477 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362480 5433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362482 5433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.362490 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363059 5433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363065 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363068 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363071 5433 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363073 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363076 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:16.366045 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363079 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363081 5433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363085 5433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363087 5433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363090 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363092 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363097 5433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363102 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363105 5433 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363108 5433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363110 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363113 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363116 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363119 5433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363121 5433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363124 5433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363126 5433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363129 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363132 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:16.366516 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363134 5433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363137 5433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363139 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363142 5433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363144 5433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363147 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363155 5433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363158 5433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363161 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363164 5433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363166 5433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363169 5433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363175 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363177 5433 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363180 5433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363183 5433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363186 5433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363188 5433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363191 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363193 5433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:16.366968 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363196 5433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363199 5433 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363201 5433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363204 5433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363206 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363209 5433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363212 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363214 5433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363217 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363219 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363222 5433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363225 5433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363227 5433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363230 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363233 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363235 5433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363238 5433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363240 5433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363243 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:16.367505 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363251 5433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363254 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363257 5433 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363259 5433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363261 5433 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363266 5433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363269 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363272 5433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363274 5433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363277 5433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363280 5433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363282 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363285 5433 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363288 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363290 5433 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363292 5433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363295 5433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363298 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363302 5433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363305 5433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:16.367943 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363315 5433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363320 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363323 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363325 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363328 5433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363331 5433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363334 5433 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363336 5433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363339 5433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363341 5433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363344 5433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363347 5433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363350 5433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363359 5433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363361 5433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363364 5433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363367 5433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363371 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363373 5433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363376 5433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:16.368429 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363378 5433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.363381 5433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363481 5433 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363493 5433 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363502 5433 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363506 5433 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363510 5433 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363514 5433 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363518 5433 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363526 5433 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363530 5433 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363533 5433 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363536 5433 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363540 5433 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363544 5433 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363547 5433 flags.go:64] FLAG: --cgroup-root="" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363550 5433 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363553 5433 flags.go:64] FLAG: --client-ca-file="" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363556 5433 flags.go:64] FLAG: --cloud-provider="external" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363559 5433 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363575 5433 flags.go:64] FLAG: --cluster-domain="" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363578 5433 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363581 5433 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363584 5433 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 14:58:16.369030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363588 5433 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363596 5433 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363605 5433 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363608 5433 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363612 5433 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363616 5433 flags.go:64] FLAG: --contention-profiling="false" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363620 5433 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363623 5433 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363626 5433 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363629 5433 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363634 5433 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363637 5433 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363640 5433 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363643 5433 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363646 5433 flags.go:64] FLAG: --enable-server="true" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363650 5433 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363656 5433 flags.go:64] FLAG: --event-burst="100" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363660 5433 flags.go:64] FLAG: --event-qps="50" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363663 5433 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363666 5433 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363669 5433 flags.go:64] FLAG: --eviction-hard="" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363673 5433 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363676 5433 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363680 5433 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363683 5433 flags.go:64] FLAG: --eviction-soft="" Nov 02 14:58:16.369960 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363686 5433 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363689 5433 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363692 5433 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363695 5433 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363698 5433 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363701 5433 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363704 5433 flags.go:64] FLAG: --feature-gates="" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363708 5433 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363711 5433 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363714 5433 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363723 5433 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363727 5433 flags.go:64] FLAG: --healthz-port="10248" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363731 5433 flags.go:64] FLAG: --help="false" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363734 5433 flags.go:64] FLAG: --hostname-override="ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363737 5433 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363741 5433 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363744 5433 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363749 5433 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363753 5433 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363755 5433 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363758 5433 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363761 5433 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363765 5433 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 14:58:16.370742 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363768 5433 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363771 5433 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363774 5433 flags.go:64] FLAG: --kube-reserved="" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363777 5433 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363780 5433 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363783 5433 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363786 5433 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363795 5433 flags.go:64] FLAG: --lock-file="" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363798 5433 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363801 5433 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363804 5433 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363810 5433 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363812 5433 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363815 5433 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363818 5433 flags.go:64] FLAG: --logging-format="text" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363821 5433 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363825 5433 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363828 5433 flags.go:64] FLAG: --manifest-url="" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363831 5433 flags.go:64] FLAG: --manifest-url-header="" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363835 5433 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363844 5433 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363850 5433 flags.go:64] FLAG: --max-pods="110" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363853 5433 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363856 5433 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363859 5433 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 14:58:16.371318 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363862 5433 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363867 5433 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363870 5433 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363873 5433 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363880 5433 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363883 5433 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363886 5433 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363889 5433 flags.go:64] FLAG: --pod-cidr="" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363892 5433 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363895 5433 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363898 5433 flags.go:64] FLAG: --pods-per-core="0" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363901 5433 flags.go:64] FLAG: --port="10250" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363905 5433 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363907 5433 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ac3bb78af063003e" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363911 5433 flags.go:64] FLAG: --qos-reserved="" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363914 5433 flags.go:64] FLAG: --read-only-port="10255" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363917 5433 flags.go:64] FLAG: --register-node="true" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363920 5433 flags.go:64] FLAG: --register-with-taints="" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363924 5433 flags.go:64] FLAG: --registry-burst="10" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363926 5433 flags.go:64] FLAG: --registry-qps="5" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363929 5433 flags.go:64] FLAG: --reserved-cpus="" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363932 5433 flags.go:64] FLAG: --reserved-memory="" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363936 5433 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363939 5433 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363943 5433 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363946 5433 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 14:58:16.371917 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363948 5433 flags.go:64] FLAG: --runonce="false" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363951 5433 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363963 5433 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363966 5433 flags.go:64] FLAG: --seccomp-default="false" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363970 5433 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363985 5433 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363988 5433 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363993 5433 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.363997 5433 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364000 5433 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364003 5433 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364006 5433 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364017 5433 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364020 5433 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364023 5433 flags.go:64] FLAG: --system-cgroups="" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364026 5433 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364031 5433 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364035 5433 flags.go:64] FLAG: --tls-cert-file="" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364038 5433 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364051 5433 flags.go:64] FLAG: --tls-min-version="" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364054 5433 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364057 5433 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364060 5433 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364063 5433 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364066 5433 flags.go:64] FLAG: --v="2" Nov 02 14:58:16.372560 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364070 5433 flags.go:64] FLAG: --version="false" Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364074 5433 flags.go:64] FLAG: --vmodule="" Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364078 5433 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364081 5433 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364276 5433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364280 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364284 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364286 5433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364289 5433 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364293 5433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364301 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364304 5433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364306 5433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364309 5433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364313 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364316 5433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364318 5433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364321 5433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364324 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364326 5433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:16.373156 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364329 5433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364331 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364334 5433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364336 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364339 5433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364341 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364344 5433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364346 5433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364349 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364352 5433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364357 5433 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364359 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364362 5433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364364 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364367 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364370 5433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364372 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364375 5433 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364377 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364380 5433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:16.373624 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364383 5433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364385 5433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364388 5433 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364396 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364399 5433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364402 5433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364406 5433 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364409 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364411 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364414 5433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364416 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364419 5433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364421 5433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364424 5433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364427 5433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364429 5433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364432 5433 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364434 5433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364437 5433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364439 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:16.374109 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364442 5433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364444 5433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364447 5433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364450 5433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364452 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364455 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364457 5433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364459 5433 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364462 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364465 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364467 5433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364470 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364473 5433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364475 5433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364478 5433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364480 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364489 5433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364491 5433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364495 5433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364498 5433 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:16.374596 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364500 5433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364503 5433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364505 5433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364508 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364511 5433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364513 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364516 5433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364519 5433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364521 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364524 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364527 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364529 5433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364532 5433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364535 5433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364537 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364540 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364542 5433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364546 5433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364550 5433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:16.375073 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364553 5433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364556 5433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364559 5433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364563 5433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364567 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364570 5433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364573 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364576 5433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364579 5433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364581 5433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364590 5433 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.364618 5433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364659 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364663 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364666 5433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:16.375770 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364668 5433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364671 5433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364674 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364676 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364679 5433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364683 5433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364687 5433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364690 5433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364693 5433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364696 5433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364700 5433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364703 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364706 5433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364709 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364712 5433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364715 5433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364717 5433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364720 5433 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:16.376298 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364723 5433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364727 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364729 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364732 5433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364735 5433 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364737 5433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364740 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364743 5433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364745 5433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364748 5433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364760 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364763 5433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364766 5433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364769 5433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364771 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364774 5433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364776 5433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364779 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364781 5433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:16.376758 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364784 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364787 5433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364789 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364792 5433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364794 5433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364797 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364799 5433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364802 5433 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364804 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364807 5433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364810 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364812 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364815 5433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364817 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364821 5433 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364824 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364827 5433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364829 5433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364832 5433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364834 5433 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:16.377418 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364837 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364840 5433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364842 5433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364846 5433 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364854 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364857 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364860 5433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364863 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364865 5433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364868 5433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364871 5433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364874 5433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364876 5433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364879 5433 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364882 5433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364885 5433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364887 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364890 5433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364892 5433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364895 5433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:16.378032 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364897 5433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364900 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364902 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364905 5433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364913 5433 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364917 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364921 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364923 5433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364926 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364928 5433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364931 5433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364934 5433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364936 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364939 5433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364941 5433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364945 5433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364948 5433 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364956 5433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364959 5433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364961 5433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:16.378506 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364964 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364967 5433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364969 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364984 5433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364986 5433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.364989 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.370677 5433 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.370695 5433 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.370719 5433 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.370724 5433 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370789 5433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370794 5433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370798 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370802 5433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370805 5433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370808 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370811 5433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370813 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370816 5433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:16.379067 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370819 5433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370822 5433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370825 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370827 5433 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370830 5433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370833 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370836 5433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370838 5433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370841 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370844 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370846 5433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370849 5433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370852 5433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370855 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370857 5433 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370860 5433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370863 5433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370865 5433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370868 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370872 5433 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:16.379705 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370875 5433 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370878 5433 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370881 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370883 5433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370886 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370889 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370892 5433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370895 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370897 5433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370900 5433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370903 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370905 5433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370908 5433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370910 5433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370913 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370915 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370918 5433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370920 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370923 5433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:16.380456 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370926 5433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370928 5433 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370931 5433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370934 5433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370937 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370939 5433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370942 5433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370944 5433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370948 5433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370952 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370955 5433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370959 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370962 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370965 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370967 5433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370990 5433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370995 5433 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.370999 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371004 5433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:16.381064 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371008 5433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371011 5433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371013 5433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371016 5433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371019 5433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371021 5433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371024 5433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371027 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371030 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371033 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371035 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371038 5433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371051 5433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371054 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371057 5433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371060 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371064 5433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371068 5433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371072 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:16.381518 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371075 5433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371078 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371080 5433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371083 5433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371086 5433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371089 5433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371091 5433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371094 5433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371097 5433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371099 5433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371103 5433 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371105 5433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371108 5433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371111 5433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371114 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371116 5433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371119 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371122 5433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371124 5433 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:16.381997 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371127 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.371156 5433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371298 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371303 5433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371307 5433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371317 5433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371321 5433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371329 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371333 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371336 5433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371339 5433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371342 5433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371344 5433 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371347 5433 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371350 5433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371352 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:16.382477 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371355 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371358 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371361 5433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371364 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371367 5433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371369 5433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371372 5433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371375 5433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371378 5433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371380 5433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371383 5433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371386 5433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371388 5433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371391 5433 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371394 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371396 5433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371398 5433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371401 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371404 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371407 5433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:16.382852 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371409 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371411 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371414 5433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371417 5433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371424 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371427 5433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371429 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371432 5433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371434 5433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371437 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371440 5433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371442 5433 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371445 5433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371447 5433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371450 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371453 5433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371456 5433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371459 5433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371461 5433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:16.383373 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371464 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371466 5433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371469 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371471 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371474 5433 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371477 5433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371479 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371482 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371484 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371487 5433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371489 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371492 5433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371494 5433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371497 5433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371500 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371502 5433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371505 5433 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371507 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371514 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371517 5433 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:16.383994 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371520 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371522 5433 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371525 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371527 5433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371532 5433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371536 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371539 5433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371541 5433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371544 5433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371547 5433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371549 5433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371552 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371554 5433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371557 5433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371560 5433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371562 5433 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371565 5433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371567 5433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371570 5433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371573 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:16.384611 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371575 5433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371578 5433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371582 5433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371586 5433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371589 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371593 5433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371596 5433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371599 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371602 5433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371604 5433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371607 5433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371618 5433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371621 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.371644 5433 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371681 5433 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:16.385291 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371685 5433 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371688 5433 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371693 5433 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371695 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371698 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371701 5433 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371703 5433 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371706 5433 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371709 5433 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371712 5433 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371715 5433 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371717 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371720 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371723 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371725 5433 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371728 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371730 5433 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371733 5433 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371736 5433 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371738 5433 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:16.385798 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371741 5433 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371744 5433 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371746 5433 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371749 5433 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371751 5433 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371754 5433 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371756 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371759 5433 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371761 5433 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371769 5433 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371772 5433 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371775 5433 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371778 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371782 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371786 5433 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371789 5433 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371791 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371794 5433 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371796 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:16.386408 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371799 5433 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371802 5433 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371804 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371807 5433 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371810 5433 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371812 5433 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371815 5433 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371818 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371820 5433 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371823 5433 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371825 5433 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371828 5433 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371830 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371833 5433 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371836 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371838 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371841 5433 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371843 5433 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371845 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:16.386863 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371848 5433 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371851 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371853 5433 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371856 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371863 5433 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371866 5433 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371870 5433 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371874 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371878 5433 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371880 5433 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371883 5433 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371885 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371888 5433 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371890 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371893 5433 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371896 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371898 5433 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371901 5433 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371903 5433 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371906 5433 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:16.387681 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371908 5433 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371911 5433 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371913 5433 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371916 5433 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371919 5433 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371922 5433 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371924 5433 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371927 5433 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371930 5433 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371932 5433 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371935 5433 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371938 5433 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371940 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371943 5433 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371945 5433 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371949 5433 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371953 5433 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371962 5433 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371965 5433 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:16.388465 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371983 5433 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371989 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371992 5433 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371994 5433 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.371997 5433 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.372000 5433 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.372002 5433 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: W1102 14:58:16.372006 5433 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.372087 5433 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.373106 5433 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.373156 5433 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.373347 5433 server.go:1014] "Starting client certificate rotation" Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.373508 5433 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.373560 5433 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:52:04 +0000 UTC" deadline="2028-05-09 13:51:01.034572318 +0000 UTC" Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.373601 5433 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13294h52m44.660973928s" Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.373927 5433 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:58:16.389017 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.375109 5433 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:58:16.389451 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.377225 5433 log.go:25] "Validated CRI v1 runtime API" Nov 02 14:58:16.389451 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.379385 5433 log.go:25] "Validated CRI v1 image API" Nov 02 14:58:16.389451 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.380315 5433 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 14:58:16.389451 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.382497 5433 fs.go:135] Filesystem UUIDs: map[022ea997-829b-4a43-ba1b-99028c13e25c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 80488794-8acc-48f4-82ba-39a0f30dedd5:/dev/nvme0n1p4] Nov 02 14:58:16.389580 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.382514 5433 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/064e024e7af9f230b540bb3051a055b92cd7bb0148e04d749780600bd7be8bf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/064e024e7af9f230b540bb3051a055b92cd7bb0148e04d749780600bd7be8bf4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07acbdd0495d2ebd0aeac4de1d2fb585afafe5f17bd5420c7eb7cfced00df472/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07acbdd0495d2ebd0aeac4de1d2fb585afafe5f17bd5420c7eb7cfced00df472/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/109c1a32d0a0161252216f1d08997d3ba0d57b1706f1104181079d0c9ee20329/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/109c1a32d0a0161252216f1d08997d3ba0d57b1706f1104181079d0c9ee20329/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2efad5608fa771bcbbb1e1c0d5f8ffc5ff8e8ba88284fc22b365b15408d69338/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2efad5608fa771bcbbb1e1c0d5f8ffc5ff8e8ba88284fc22b365b15408d69338/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fc621fd8ae7d5e63ee52805d29176009c9c425652cc3f205650c69783e5f9e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fc621fd8ae7d5e63ee52805d29176009c9c425652cc3f205650c69783e5f9e8/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fdade16c0d70ce01cd2a8280a9d25215113305d0f78e6caafa7938ae784b9f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fdade16c0d70ce01cd2a8280a9d25215113305d0f78e6caafa7938ae784b9f4/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33caa47232f121ab61096a4b20f93a65e9605f8dd64cc61870fe44857d734578/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33caa47232f121ab61096a4b20f93a65e9605f8dd64cc61870fe44857d734578/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46ea21093ca88ee225361b61733b6c3e59e045dcaf8bf3d6de3d3d832491fcac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46ea21093ca88ee225361b61733b6c3e59e045dcaf8bf3d6de3d3d832491fcac/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cd2d0008bd4daadc62312915a67f350ebd6850b885725020e29208c6219deee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cd2d0008bd4daadc62312915a67f350ebd6850b885725020e29208c6219deee/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/539399b817160e8f3020c6fc70c3ce2c558700e8f63f51be00e40c9ebcaab21a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/539399b817160e8f3020c6fc70c3ce2c558700e8f63f51be00e40c9ebcaab21a/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c7a75b887a2f15f14e1e7f1b82ba9dcafda77d8ff0717a53f87d05aa32cf2e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c7a75b887a2f15f14e1e7f1b82ba9dcafda77d8ff0717a53f87d05aa32cf2e5/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9617a9080a2344017c32fbbe264d351f4023d7f09c96491cb5cfa2687752714a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9617a9080a2344017c32fbbe264d351f4023d7f09c96491cb5cfa2687752714a/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e63a570836e0675f700db3e145705fc4b1a2d5f2af7361e8081175f513adb10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e63a570836e0675f700db3e145705fc4b1a2d5f2af7361e8081175f513adb10/userdata/shm major:0 minor:388 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5c71fadb1c1bc9e1fc6f904ccea9974225acd46a547316a4b537794092e5c12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5c71fadb1c1bc9e1fc6f904ccea9974225acd46a547316a4b537794092e5c12/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a87a11243e7dcbadb85ed7f01766cd29da43e41dba0bb095536e0c3fff77d794/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a87a11243e7dcbadb85ed7f01766cd29da43e41dba0bb095536e0c3fff77d794/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2e1eda306ef00dba83f3cfbc19161636503b8132a5c04c5f2c5390235130f1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2e1eda306ef00dba83f3cfbc19161636503b8132a5c04c5f2c5390235130f1d/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c10c3fc3c2de2e1822ec788cb9e97f58911c8a5559f1b73b0ffcdedf49328414/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c10c3fc3c2de2e1822ec788cb9e97f58911c8a5559f1b73b0ffcdedf49328414/userdata/shm major:0 minor:383 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6ad51e734f9ef12ffa6eb9eef9a56b2b3e4aa191e810365072c35563dbdda99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6ad51e734f9ef12ffa6eb9eef9a56b2b3e4aa191e810365072c35563dbdda99/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc5cf81c7d5fe600c530c87d583076f795c8fd1e528987c6da619cc71eb113ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc5cf81c7d5fe600c530c87d583076f795c8fd1e528987c6da619cc71eb113ac/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdcdb3a8d3ecfa711d4d7abc02ea3a685040982f19b5170110b6a82035e5c337/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdcdb3a8d3ecfa711d4d7abc02ea3a685040982f19b5170110b6a82035e5c337/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdd0264f39a453d8b6ed93bd8362b977e34fd8d43d77989f7a7ca45abe9e0bfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdd0264f39a453d8b6ed93bd8362b977e34fd8d43d77989f7a7ca45abe9e0bfc/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8fbbd7dcba1878386c62ef3067fb92cd8b2be038bb29181b570d5610df31e48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8fbbd7dcba1878386c62ef3067fb92cd8b2be038bb29181b570d5610df31e48/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2616b3d7-11af-4008-99e7-44a5c575e85a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2616b3d7-11af-4008-99e7-44a5c575e85a/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dc3e10d-fde4-4b8a-97b7-e065848bed64/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2dc3e10d-fde4-4b8a-97b7-e065848bed64/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dc3e10d-fde4-4b8a-97b7-e065848bed64/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2dc3e10d-fde4-4b8a-97b7-e065848bed64/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dc3e10d-fde4-4b8a-97b7-e065848bed64/volumes/kubernetes.io~projected/kube-api-access-xlfss:{mountpoint:/var/lib/kubelet/pods/2dc3e10d-fde4-4b8a-97b7-e065848bed64/volumes/kubernetes.io~projected/kube-api-access-xlfss major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6/volumes/kubernetes.io~projected/kube-api-access-phl58:{mountpoint:/var/lib/kubelet/pods/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6/volumes/kubernetes.io~projected/kube-api-access-phl58 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c7d4401-1b82-4665-8838-9e05fac70e69/volumes/kubernetes.io~projected/kube-api-access-nb969:{mountpoint:/var/lib/kubelet/pods/6c7d4401-1b82-4665-8838-9e05fac70e69/volumes/kubernetes.io~projected/kube-api-access-nb969 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c7d4401-1b82-4665-8838-9e05fac70e69/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6c7d4401-1b82-4665-8838-9e05fac70e69/volumes/kubernetes.io~secret/metrics-certs major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/795a4ecf-3ca0-4699-a06a-a051078f9fd1/volumes/kubernetes.io~projected/kube-api-access-hsqwp:{mountpoint:/var/lib/kubelet/pods/795a4ecf-3ca0-4699-a06a-a051078f9fd1/volumes/kubernetes.io~projected/kube-api-access-hsqwp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eff5e30-0f92-4736-a63f-c2a02ceb5a47/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7eff5e30-0f92-4736-a63f-c2a02ceb5a47/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eff5e30-0f92-4736-a63f-c2a02ceb5a47/volumes/kubernetes.io~projected/kube-api-access-6jvst:{mountpoint:/var/lib/kubelet/pods/7eff5e30-0f92-4736-a63f-c2a02ceb5a47/volumes/kubernetes.io~projected/kube-api-access-6jvst major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eff5e30-0f92-4736-a63f-c2a02ceb5a47/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7eff5e30-0f92-4736-a63f-c2a02ceb5a47/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80cddaf6-6206-4424-bff1-953ac57d7c81/volumes/kubernetes.io~projected/kube-api-access-6hhzc:{mountpoint:/var/lib/kubelet/pods/80cddaf6-6206-4424-bff1-953ac57d7c81/volumes/kubernetes.io~projected/kube-api-access-6hhzc major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~projected/kube-api-access-5jcrc:{mountpoint:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~projected/kube-api-access-5jcrc major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/888654bd-3ef7-4ff1-b31e-0ce4be3587b6/volumes/kubernetes.io~projected/kube-api-access-7ll25:{mountpoint:/var/lib/kubelet/pods/888654bd-3ef7-4ff1-b31e-0ce4be3587b6/volumes/kubernetes.io~projected/kube-api-access-7ll25 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/888654bd-3ef7-4ff1-b31e-0ce4be3587b6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/888654bd-3ef7-4ff1-b31e-0ce4be3587b6/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b6bfaad-2c57-4baa-be7d-f25715574792/volumes/kubernetes.io~projected/kube-api-access-4mr5q:{mountpoint:/var/lib/kubelet/pods/8b6bfaad-2c57-4baa-be7d-f25715574792/volumes/kubernetes.io~projected/kube-api-access-4mr5q major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4/volumes/kubernetes.io~projected/kube-api-access-gdzxj:{mountpoint:/var/lib/kubelet/pods/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4/volumes/kubernetes.io~projected/kube-api-access-gdzxj major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ac6fea3-ce51-474e-bb44-9b328a950fbe/volumes/kubernetes.io~projected/kube-api-access-7b6kz:{mountpoint:/var/lib/kubelet/pods/9ac6fea3-ce51-474e-bb44-9b328a950fbe/volumes/kubernetes.io~projected/kube-api-access-7b6kz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0364fe3-6710-4de9-af3b-cd66989104f8/volumes/kubernetes.io~projected/kube-api-access-q976l:{mountpoint:/var/lib/kubelet/pods/a0364fe3-6710-4de9-af3b-cd66989104f8/volumes/kubernetes.io~projected/kube-api-access-q976l major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445/volumes/kubernetes.io~projected/kube-api-access-7jtd5:{mountpoint:/var/lib/kubelet/pods/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445/volumes/kubernetes.io~projected/kube-api-access-7jtd5 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd0ae468-b91d-4121-baca-0cc3836997ae/volumes/kubernetes.io~projected/kube-api-access-bn7m4:{mountpoint:/var/lib/kubelet/pods/bd0ae468-b91d-4121-baca-0cc3836997ae/volumes/kubernetes.io~projected/kube-api-access-bn7m4 major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd0ae468-b91d-4121-baca-0cc3836997ae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bd0ae468-b91d-4121-baca-0cc3836997ae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd0ae468-b91d-4121-baca-0cc3836997ae/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/bd0ae468-b91d-4121-baca-0cc3836997ae/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4f454c8-54e1-4eba-a8ea-10a32729ce0e/volumes/kubernetes.io~projected/kube-api-access-929p7:{mountpoint:/var/lib/kubelet/pods/d4f454c8-54e1-4eba-a8ea-10a32729ce0e/volumes/kubernetes.io~projected/kube-api-access-929p7 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4f454c8-54e1-4eba-a8ea-10a32729ce0e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d4f454c8-54e1-4eba-a8ea-10a32729ce0e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~projected/kube-api-access-2hxbs:{mountpoint:/var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~projected/kube-api-access-2hxbs major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/volumes/kubernetes.io~projected/kube-api-access-96gzz:{mountpoint:/var/lib/kubelet/pods/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/volumes/kubernetes.io~projected/kube-api-access-96gzz major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9695a92-acd9-41de-bfcf-13d149ecb730/volumes/kubernetes.io~projected/kube-api-access-dsqxh:{mountpoint:/var/lib/kubelet/pods/e9695a92-acd9-41de-bfcf-13d149ecb730/volumes/kubernetes.io~projected/kube-api-access-dsqxh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/volumes/kubernetes.io~projected/kube-api-access-w7dn7:{mountpoint:/var/lib/kubelet/pods/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/volumes/kubernetes.io~projected/kube-api-access-w7dn7 major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed84914e-49d4-4f14-877b-ca1b0cdff144/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ed84914e-49d4-4f14-877b-ca1b0cdff144/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:382 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/02f274e0874baa4fad1a593a1862f6f561be753c704255aad26261e524cf63a0/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/90f78bfcbc0334743eebb7ce68d2a9c96f681fb271ac2548a04fd50b0dcf0a73/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d1e04108af1ce5e1f5ca62f3eb1c8e9369754a048c882f57b86deddf6f37920f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/aab67ee06cfee18fb7e10fc39647f22690de849f9fdf1e6998ecd0eb602df44d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/4227f710222b66f481dd88cb5a3a9ac473261355a8f501cedf9e5c635dc6fe07/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/47e58fae77569e40ab6bf8a9cc4c7456a5b25f841379fc5eb3bf27cc7d71773d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/244d71ed1f1c47a452051354bf374879f764904801d0d6303b95b8bcf3e9d55d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8bc1d0322a0f931dc506a56371754efd3aba97674304bd7ff066cfba65514302/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8d5daf561de9386c475a993903ee1ee96b175b741a604e60125224894e586c59/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/748d2a65736fcf660d62f122f03ca87ad4bd313ef5e4e5ba2721272307c2ca5d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/3cda780549d78b91e5fc23551712d6484c915430d10ac7c6141f62ab6cf39f27/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/62ef76f7a1a8068decc15c0389bcb49e2d23b89811a16641c2fb14e487778cf5/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/59b60b3458cbf5df593868c0ac407a2da933c058340d3310e5aaad533e2a4c0c/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2583a7d8c5a44a1a0233d49e7a8b0fceafe75797755e7fa0fcac2eae1ca0b79e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9ab9fd333bec27884df7c44ecdd34a4fed9588d7c52bbe879ab35b39e3e65b00/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d2935cd7664702cd5ec56808a59004f46ae7d30d9411001d4461c53971b736a2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4bf7d6e1d2f4f5bed95b41c556a5579d07e77ff82168025c348734021cfc446f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9da67fcbc5aa2f3deb71dc32e98f94d2acfd814bf96bb1c59fd80159f337a309/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1b6e2600e3a0dff89003950034bfc7ce556111142e38a9d2dfd7fbbcbd1da064/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/040c31286336107e2cada13a28a061a7c32e58ec7f966e639abaefc2c8ebc319/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3b6e81ab5c87a57e4f42dab6aa96dcc811b0239f44112ba64cb5a03d6dc7581e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f63d79568febfeb21ecda5c8f1fa77c3b10a992e2b2556e2f3562bb7c64abed8/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/3fb79702cd2ea9e27a241fd6e3847d759eadd2a8753b6032dd44661f10286e48/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d4ce9e4b144e54975ea48850bf1956f91c89646f6c94bdcb3782deb3854afe1c/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/b96dec9984f8d6ab42d0fe747274aec9b1ba2c0ed5585b757250fb804220b02f/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/2b0a58a7ea8356f1385d59c1c4e683764dd691e34ae37e3b6213bf510d6fefad/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/b6d1dae3a8231a471bb75b5258611d261004caf4abc4332d71f4a63cddcc2da1/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/b6b1a069da9699b417e95a60f7f620da96d8868c54afc22c6e0ea9fae4881240/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/dcf253374b87f66f2fa6b2acf5d9e8238c10ceada0491c5639c999de43cb4297/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/3dd5c884cab59e071235f66a54d97c4601c9f6fd00a863420de5ebcc9b5dee8d/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/da1f66a011016b6231f54924f681f3123014ae565195a8711a4d75b743bc7263/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/45bfae467e3739995a0552b91abff5f4a7d2bc82336a5bc6fafda23f4eff1670/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/d373708fa6dfbe4a48524d16739295d5639def085195520a30d7c36a464913bf/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/d4b20e6610ebfa967d5c911a99ab4ed392a5405300b7ee59494b84bc0c0b6616/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/761f12fc58667199663658b712a489e7bce8b814a69cd4a3bc040d0eb88caf73/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/5156cbea2d2978575f69179edbe329c459e3a1832177c1d9619dac460c48358f/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/7e877c8fc34f164611a24162c0b4d5fac10c268a05932360fa146743e97a2b17/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/df1df1e899f39bb9d4b1125d90554fbf3d5466b4cc440ba2c5eb128934a1a4ef/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/66ec1cfd974f05ce86cfc85281d089814e5f685e073194360f11139ff006e998/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a9cc3010091e670fea3923ca4e7f8cf2fbe63fd8cc7143afc23fe5376d094e10/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/c0be110d136d9b740e70460f5ab567d1a14d1a97df85c5b1621c6d621cda39bf/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/4b9b4c5ac9b43b9892d27071bd6c433899aab74848a50fadb1b0b6dd00323fa8/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/38055905d78e3f87f81b1ea397ba12dc39897b6d785b98bf42166e1beb0ae854/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/46c19083ad1091c56a38b91e821d705ff8b89fc5c9b56943626cd605ad69e839/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/3b09d559fc471ed203c64c0aa617bb4bea92e472f31945a798db68f54d62db69/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/65b7f01d4f102125e50c1f99daa131cc44f91239d4886c3341da00684e40c46e/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/248ea1ed42913d26368980d68f4863ac82fb02761d32b199bcef7ba8990b4217/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/4190ba9bb415dac9f2880ef3ad1e9f88befea62796b8f4ff6e135b5f62d65c2c/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/87bacc95b38fea54bc2a8247a4f2ac6e9c7616b6a43d18d2c7453a8c6b8c710f/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/d30c8e36aa1fb8ee2ffa45dd5d4c10c228b57f97cf031d31f2a973a04034b5d0/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/3c573d0500fb3c70457d3a99eac37b6fd199f1785cea7377bcb39d9dbe05f856/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/3a9bc7c78c0b958c5f50946c4f7ca989a9c0effd15ef2820e1750230b735347e/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/a4a5dc71b7f7be6233d29d3cdac805a6d3d19b7241ab203511acefa781ec4264/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/a3789b039c45028e46c4e1ee05dcd7675809bb09e7accbbf40c8e97abd8c4a9d/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/4f7b4a7cd9818d0217f06925421ce1e0085abce8bcad95b872dc8a3f584eb396/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/426e541dbee01efca3f2adde8dd8e4be8b0dd0867e24e9625de7a9ff7358eb47/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/5539e7fbaa8b9fc0c1ca3a1e7130ec6a6802fcc38556ab3a0f9f42f92d5aa3e7/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/bd8963662da0bd845e2092462c5c39bacc0f25082fd012202ef9f1e0c2c1d551/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/08c90a7cf83a4769b899b47803c5b6ae5ccc95d301d683fe5cefc81c14e95d94/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b89f7d73e40325c2a74ec0290b0fdc89c37c5e1fa0c6a1ae97e1d992aaa5c338/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2f06986122d60595c1a68d4dc8f5914c292f1cf615683fdb178acbcbedbd0634/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/5497ba4c421ee8351de3b5956b26c4af166742b41804261c868830ffdb7e52ad/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/43839afaddc8167a325e8ff01b05c57df6e04fa4879de1646fc18b5655518e1a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/7935ba324e73ca1af3d3c4bf287030ac77ff96858ed4d88925be2679401998f5/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/65202aaaa22013c96f36638671cf0882687e7cb226f7619621924f5e6358caac/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/867232abf79aa58ed1e4d0520dbc2a0dbb771344a21f0a3f3ebf6ce9671dc3cd/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/e8913ded3d9242c0462ea49c12f6552cacd2a710375824bb1e96d5069c366f86/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/e5f00a282539d0186dbce2fa92257efe58a9efc918f20548260acb6a6036ceb3/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/e8b7a09c57112e73c10320e31ed692583bbca7423408fe1d2d670be2a1739acf/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 02 14:58:16.392283 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.391895 5433 manager.go:217] Machine: {Timestamp:2026-11-02 14:58:16.391421944 +0000 UTC m=+0.064625131 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103138 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c34a9bf8814896a57e21359c25744 SystemUUID:ec2c34a9-bf88-1489-6a57-e21359c25744 BootID:df614fbc-08c7-48f2-917a-47bd3bd47427 Filesystems:[{Device:/var/lib/kubelet/pods/9ac6fea3-ce51-474e-bb44-9b328a950fbe/volumes/kubernetes.io~projected/kube-api-access-7b6kz DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9617a9080a2344017c32fbbe264d351f4023d7f09c96491cb5cfa2687752714a/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~projected/kube-api-access-2hxbs DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eff5e30-0f92-4736-a63f-c2a02ceb5a47/volumes/kubernetes.io~projected/kube-api-access-6jvst DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c7a75b887a2f15f14e1e7f1b82ba9dcafda77d8ff0717a53f87d05aa32cf2e5/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~projected/kube-api-access-5jcrc DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33caa47232f121ab61096a4b20f93a65e9605f8dd64cc61870fe44857d734578/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a87a11243e7dcbadb85ed7f01766cd29da43e41dba0bb095536e0c3fff77d794/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445/volumes/kubernetes.io~projected/kube-api-access-7jtd5 DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0364fe3-6710-4de9-af3b-cd66989104f8/volumes/kubernetes.io~projected/kube-api-access-q976l DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46ea21093ca88ee225361b61733b6c3e59e045dcaf8bf3d6de3d3d832491fcac/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8fbbd7dcba1878386c62ef3067fb92cd8b2be038bb29181b570d5610df31e48/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:306 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c7d4401-1b82-4665-8838-9e05fac70e69/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2dc3e10d-fde4-4b8a-97b7-e065848bed64/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/109c1a32d0a0161252216f1d08997d3ba0d57b1706f1104181079d0c9ee20329/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4/volumes/kubernetes.io~projected/kube-api-access-gdzxj DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c7d4401-1b82-4665-8838-9e05fac70e69/volumes/kubernetes.io~projected/kube-api-access-nb969 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc5cf81c7d5fe600c530c87d583076f795c8fd1e528987c6da619cc71eb113ac/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/volumes/kubernetes.io~projected/kube-api-access-w7dn7 DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4f454c8-54e1-4eba-a8ea-10a32729ce0e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/064e024e7af9f230b540bb3051a055b92cd7bb0148e04d749780600bd7be8bf4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cd2d0008bd4daadc62312915a67f350ebd6850b885725020e29208c6219deee/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4f454c8-54e1-4eba-a8ea-10a32729ce0e/volumes/kubernetes.io~projected/kube-api-access-929p7 DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2616b3d7-11af-4008-99e7-44a5c575e85a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6ad51e734f9ef12ffa6eb9eef9a56b2b3e4aa191e810365072c35563dbdda99/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eff5e30-0f92-4736-a63f-c2a02ceb5a47/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/795a4ecf-3ca0-4699-a06a-a051078f9fd1/volumes/kubernetes.io~projected/kube-api-access-hsqwp DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80cddaf6-6206-4424-bff1-953ac57d7c81/volumes/kubernetes.io~projected/kube-api-access-6hhzc DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd0ae468-b91d-4121-baca-0cc3836997ae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6/volumes/kubernetes.io~projected/kube-api-access-phl58 DeviceMajor:0 DeviceMinor:325 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07acbdd0495d2ebd0aeac4de1d2fb585afafe5f17bd5420c7eb7cfced00df472/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/888654bd-3ef7-4ff1-b31e-0ce4be3587b6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5c71fadb1c1bc9e1fc6f904ccea9974225acd46a547316a4b537794092e5c12/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eff5e30-0f92-4736-a63f-c2a02ceb5a47/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdd0264f39a453d8b6ed93bd8362b977e34fd8d43d77989f7a7ca45abe9e0bfc/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fdade16c0d70ce01cd2a8280a9d25215113305d0f78e6caafa7938ae784b9f4/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2efad5608fa771bcbbb1e1c0d5f8ffc5ff8e8ba88284fc22b365b15408d69338/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2e1eda306ef00dba83f3cfbc19161636503b8132a5c04c5f2c5390235130f1d/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83a2eda6-8157-428a-b0f3-c2cc77563252/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:308 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2dc3e10d-fde4-4b8a-97b7-e065848bed64/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8b6bfaad-2c57-4baa-be7d-f25715574792/volumes/kubernetes.io~projected/kube-api-access-4mr5q DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/888654bd-3ef7-4ff1-b31e-0ce4be3587b6/volumes/kubernetes.io~projected/kube-api-access-7ll25 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd0ae468-b91d-4121-baca-0cc3836997ae/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd0ae468-b91d-4121-baca-0cc3836997ae/volumes/kubernetes.io~projected/kube-api-access-bn7m4 DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdcdb3a8d3ecfa711d4d7abc02ea3a685040982f19b5170110b6a82035e5c337/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fc621fd8ae7d5e63ee52805d29176009c9c425652cc3f205650c69783e5f9e8/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2dc3e10d-fde4-4b8a-97b7-e065848bed64/volumes/kubernetes.io~projected/kube-api-access-xlfss DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9695a92-acd9-41de-bfcf-13d149ecb730/volumes/kubernetes.io~projected/kube-api-access-dsqxh DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/539399b817160e8f3020c6fc70c3ce2c558700e8f63f51be00e40c9ebcaab21a/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e63a570836e0675f700db3e145705fc4b1a2d5f2af7361e8081175f513adb10/userdata/shm DeviceMajor:0 DeviceMinor:388 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:247 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/volumes/kubernetes.io~projected/kube-api-access-96gzz DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e30a530f-5e3c-48f7-8872-f0b58d08a91e/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c10c3fc3c2de2e1822ec788cb9e97f58911c8a5559f1b73b0ffcdedf49328414/userdata/shm DeviceMajor:0 DeviceMinor:383 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed84914e-49d4-4f14-877b-ca1b0cdff144/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:07acbdd0495d2eb MacAddress:42:af:03:83:f9:31 Speed:10000 Mtu:8901} {Name:109c1a32d0a0161 MacAddress:fe:1e:5a:1c:d1:34 Speed:10000 Mtu:8901} {Name:2efad5608fa771b MacAddress:6e:ea:ce:c6:ed:5a Speed:10000 Mtu:8901} {Name:539399b817160e8 MacAddress:f6:73:ce:37:8e:b3 Speed:10000 Mtu:8901} {Name:9617a9080a23440 MacAddress:ca:7d:f1:12:ff:af Speed:10000 Mtu:8901} {Name:9e63a570836e067 MacAddress:da:7e:8c:20:bc:16 Speed:10000 Mtu:8901} {Name:b2e1eda306ef00d MacAddress:36:94:64:77:3c:95 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e8:76:c9:ff Speed:0 Mtu:9001} {Name:br-int MacAddress:da:17:72:d5:de:9f Speed:0 Mtu:8901} {Name:c10c3fc3c2de2e1 MacAddress:92:4a:13:b8:d1:35 Speed:10000 Mtu:8901} {Name:c6ad51e734f9ef1 MacAddress:7a:03:23:7f:e7:56 Speed:10000 Mtu:8901} {Name:cdcdb3a8d3ecfa7 MacAddress:fe:d9:15:4a:46:f4 Speed:10000 Mtu:8901} {Name:cdd0264f39a453d MacAddress:36:c6:17:c1:18:aa Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e8:76:c9:ff Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:32:86:92:2d:b2:90 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e6:f1:8e:b0:61:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 14:58:16.392510 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.392287 5433 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 14:58:16.392510 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.392344 5433 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 14:58:16.392727 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.392703 5433 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 14:58:16.392861 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.392728 5433 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-181.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 14:58:16.392904 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.392864 5433 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 14:58:16.392904 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.392871 5433 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 14:58:16.392904 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.392881 5433 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:58:16.392904 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.392901 5433 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:58:16.393099 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.393011 5433 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 14:58:16.393194 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.393184 5433 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 14:58:16.393306 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.393298 5433 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 14:58:16.393340 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.393333 5433 kubelet.go:493] "Attempting to sync node with API server" Nov 02 14:58:16.393371 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.393342 5433 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 14:58:16.393371 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.393353 5433 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 14:58:16.393371 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.393362 5433 kubelet.go:405] "Adding apiserver pod source" Nov 02 14:58:16.393484 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.393379 5433 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 14:58:16.394176 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.394160 5433 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 14:58:16.394223 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.394178 5433 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:58:16.394779 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.394757 5433 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 14:58:16.398848 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.398831 5433 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 14:58:16.398848 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.398845 5433 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:58:16.398965 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.398955 5433 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 14:58:16.399530 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399517 5433 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 14:58:16.399561 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399548 5433 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 14:58:16.399642 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399634 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 14:58:16.399678 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399648 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 14:58:16.399678 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399657 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 14:58:16.399678 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399666 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 14:58:16.399678 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399672 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 14:58:16.399678 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399678 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 14:58:16.399810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399684 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 14:58:16.399810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399690 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 14:58:16.399810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399697 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 14:58:16.399810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399703 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 14:58:16.399810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399714 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 14:58:16.399810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399728 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 14:58:16.399810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399758 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 14:58:16.399810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.399768 5433 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 14:58:16.403233 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.403217 5433 server.go:1263] "Started kubelet" Nov 02 14:58:16.403332 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.403286 5433 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 14:58:16.404104 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.403445 5433 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 14:58:16.404104 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.403776 5433 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 14:58:16.403807 ip-10-0-135-181 systemd[1]: Started Kubernetes Kubelet. Nov 02 14:58:16.404688 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.404643 5433 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 14:58:16.410032 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.406339 5433 server.go:317] "Adding debug handlers to kubelet server" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.410608 5433 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.410626 5433 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.410669 5433 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:52:06 +0000 UTC" deadline="2028-04-12 21:09:10.137710571 +0000 UTC" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.410688 5433 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12654h10m53.727025728s" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.410720 5433 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.410728 5433 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:16.410851 5433 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.410914 5433 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413212 5433 factory.go:55] Registering systemd factory Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413235 5433 factory.go:223] Registration of the systemd container factory successfully Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413478 5433 factory.go:153] Registering CRI-O factory Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413494 5433 factory.go:223] Registration of the crio container factory successfully Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413505 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a2eda6-8157-428a-b0f3-c2cc77563252" volumeName="kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413537 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a2eda6-8157-428a-b0f3-c2cc77563252" volumeName="kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413549 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b6bfaad-2c57-4baa-be7d-f25715574792" volumeName="kubernetes.io/projected/8b6bfaad-2c57-4baa-be7d-f25715574792-kube-api-access-4mr5q" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413557 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9695a92-acd9-41de-bfcf-13d149ecb730" volumeName="kubernetes.io/empty-dir/e9695a92-acd9-41de-bfcf-13d149ecb730-tmp-dir" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413565 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f" volumeName="kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-tls" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413573 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6" volumeName="kubernetes.io/secret/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413581 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eff5e30-0f92-4736-a63f-c2a02ceb5a47" volumeName="kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-env-overrides" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413589 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a2eda6-8157-428a-b0f3-c2cc77563252" volumeName="kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413598 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="888654bd-3ef7-4ff1-b31e-0ce4be3587b6" volumeName="kubernetes.io/secret/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-cert" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413541 5433 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413631 5433 factory.go:103] Registering Raw factory Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413643 5433 manager.go:1199] Started watching for new ooms in manager Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413605 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3" volumeName="kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413734 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f" volumeName="kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413761 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dc3e10d-fde4-4b8a-97b7-e065848bed64" volumeName="kubernetes.io/projected/2dc3e10d-fde4-4b8a-97b7-e065848bed64-kube-api-access-xlfss" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413777 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eff5e30-0f92-4736-a63f-c2a02ceb5a47" volumeName="kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-script-lib" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413791 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab7df6cc-81c7-4ecc-b1f8-4b8de2464445" volumeName="kubernetes.io/configmap/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413808 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9695a92-acd9-41de-bfcf-13d149ecb730" volumeName="kubernetes.io/projected/e9695a92-acd9-41de-bfcf-13d149ecb730-kube-api-access-dsqxh" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413822 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dc3e10d-fde4-4b8a-97b7-e065848bed64" volumeName="kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-tuned" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413836 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a2eda6-8157-428a-b0f3-c2cc77563252" volumeName="kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413849 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30a530f-5e3c-48f7-8872-f0b58d08a91e" volumeName="kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-metrics-server-audit-profiles" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413861 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30a530f-5e3c-48f7-8872-f0b58d08a91e" volumeName="kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413876 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3" volumeName="kubernetes.io/projected/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-kube-api-access-96gzz" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413889 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f" volumeName="kubernetes.io/empty-dir/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-textfile" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413919 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed84914e-49d4-4f14-877b-ca1b0cdff144" volumeName="kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413933 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a2eda6-8157-428a-b0f3-c2cc77563252" volumeName="kubernetes.io/projected/83a2eda6-8157-428a-b0f3-c2cc77563252-kube-api-access-5jcrc" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413947 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b6bfaad-2c57-4baa-be7d-f25715574792" volumeName="kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-binary-copy" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413960 5433 manager.go:319] Starting recovery of all containers Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.413959 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f454c8-54e1-4eba-a8ea-10a32729ce0e" volumeName="kubernetes.io/empty-dir/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-tmp-dir" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416530 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f454c8-54e1-4eba-a8ea-10a32729ce0e" volumeName="kubernetes.io/projected/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-kube-api-access-929p7" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416545 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dc3e10d-fde4-4b8a-97b7-e065848bed64" volumeName="kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-tmp" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416555 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6" volumeName="kubernetes.io/projected/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-kube-api-access-phl58" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416569 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a2eda6-8157-428a-b0f3-c2cc77563252" volumeName="kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416578 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4" volumeName="kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416585 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab7df6cc-81c7-4ecc-b1f8-4b8de2464445" volumeName="kubernetes.io/empty-dir/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-data-volume" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416595 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd0ae468-b91d-4121-baca-0cc3836997ae" volumeName="kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416604 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd0ae468-b91d-4121-baca-0cc3836997ae" volumeName="kubernetes.io/empty-dir/bd0ae468-b91d-4121-baca-0cc3836997ae-volume-directive-shadow" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416611 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30a530f-5e3c-48f7-8872-f0b58d08a91e" volumeName="kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416619 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c7d4401-1b82-4665-8838-9e05fac70e69" volumeName="kubernetes.io/projected/6c7d4401-1b82-4665-8838-9e05fac70e69-kube-api-access-nb969" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416627 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80cddaf6-6206-4424-bff1-953ac57d7c81" volumeName="kubernetes.io/projected/80cddaf6-6206-4424-bff1-953ac57d7c81-kube-api-access-6hhzc" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416634 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ac6fea3-ce51-474e-bb44-9b328a950fbe" volumeName="kubernetes.io/projected/9ac6fea3-ce51-474e-bb44-9b328a950fbe-kube-api-access-7b6kz" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416642 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd0ae468-b91d-4121-baca-0cc3836997ae" volumeName="kubernetes.io/projected/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-api-access-bn7m4" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416649 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30a530f-5e3c-48f7-8872-f0b58d08a91e" volumeName="kubernetes.io/projected/e30a530f-5e3c-48f7-8872-f0b58d08a91e-kube-api-access-2hxbs" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416657 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3" volumeName="kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416666 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd0ae468-b91d-4121-baca-0cc3836997ae" volumeName="kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416674 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2616b3d7-11af-4008-99e7-44a5c575e85a" volumeName="kubernetes.io/configmap/2616b3d7-11af-4008-99e7-44a5c575e85a-konnectivity-ca" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416681 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a2eda6-8157-428a-b0f3-c2cc77563252" volumeName="kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-grpc-tls" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416689 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0364fe3-6710-4de9-af3b-cd66989104f8" volumeName="kubernetes.io/configmap/a0364fe3-6710-4de9-af3b-cd66989104f8-iptables-alerter-script" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416696 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30a530f-5e3c-48f7-8872-f0b58d08a91e" volumeName="kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-tls" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416704 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a2eda6-8157-428a-b0f3-c2cc77563252" volumeName="kubernetes.io/empty-dir/83a2eda6-8157-428a-b0f3-c2cc77563252-tmp" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416713 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ac6fea3-ce51-474e-bb44-9b328a950fbe" volumeName="kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cni-binary-copy" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416720 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd0ae468-b91d-4121-baca-0cc3836997ae" volumeName="kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416728 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f" volumeName="kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-metrics-client-ca" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416741 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eff5e30-0f92-4736-a63f-c2a02ceb5a47" volumeName="kubernetes.io/projected/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-kube-api-access-6jvst" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416760 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd0ae468-b91d-4121-baca-0cc3836997ae" volumeName="kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-metrics-client-ca" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416767 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30a530f-5e3c-48f7-8872-f0b58d08a91e" volumeName="kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-client-ca-bundle" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416775 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f" volumeName="kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416784 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2616b3d7-11af-4008-99e7-44a5c575e85a" volumeName="kubernetes.io/secret/2616b3d7-11af-4008-99e7-44a5c575e85a-agent-certs" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416791 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab7df6cc-81c7-4ecc-b1f8-4b8de2464445" volumeName="kubernetes.io/secret/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416799 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f454c8-54e1-4eba-a8ea-10a32729ce0e" volumeName="kubernetes.io/configmap/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-config-volume" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416813 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f" volumeName="kubernetes.io/projected/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-kube-api-access-w7dn7" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416821 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="888654bd-3ef7-4ff1-b31e-0ce4be3587b6" volumeName="kubernetes.io/projected/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-kube-api-access-7ll25" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416828 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b6bfaad-2c57-4baa-be7d-f25715574792" volumeName="kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416835 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ac6fea3-ce51-474e-bb44-9b328a950fbe" volumeName="kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-daemon-config" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416843 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab7df6cc-81c7-4ecc-b1f8-4b8de2464445" volumeName="kubernetes.io/projected/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-api-access-7jtd5" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416851 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4f454c8-54e1-4eba-a8ea-10a32729ce0e" volumeName="kubernetes.io/secret/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-metrics-tls" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416858 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30a530f-5e3c-48f7-8872-f0b58d08a91e" volumeName="kubernetes.io/empty-dir/e30a530f-5e3c-48f7-8872-f0b58d08a91e-audit-log" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416866 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="795a4ecf-3ca0-4699-a06a-a051078f9fd1" volumeName="kubernetes.io/configmap/795a4ecf-3ca0-4699-a06a-a051078f9fd1-serviceca" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416881 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eff5e30-0f92-4736-a63f-c2a02ceb5a47" volumeName="kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-config" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416893 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eff5e30-0f92-4736-a63f-c2a02ceb5a47" volumeName="kubernetes.io/secret/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416903 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a2eda6-8157-428a-b0f3-c2cc77563252" volumeName="kubernetes.io/configmap/83a2eda6-8157-428a-b0f3-c2cc77563252-metrics-client-ca" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416911 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b6bfaad-2c57-4baa-be7d-f25715574792" volumeName="kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416918 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0364fe3-6710-4de9-af3b-cd66989104f8" volumeName="kubernetes.io/projected/a0364fe3-6710-4de9-af3b-cd66989104f8-kube-api-access-q976l" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416925 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3" volumeName="kubernetes.io/configmap/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-metrics-client-ca" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416933 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c7d4401-1b82-4665-8838-9e05fac70e69" volumeName="kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416940 5433 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="795a4ecf-3ca0-4699-a06a-a051078f9fd1" volumeName="kubernetes.io/projected/795a4ecf-3ca0-4699-a06a-a051078f9fd1-kube-api-access-hsqwp" seLinuxMountContext="" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416949 5433 reconstruct.go:97] "Volume reconstruction finished" Nov 02 14:58:16.420207 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.416954 5433 reconciler.go:29] "Reconciler: start to sync state" Nov 02 14:58:16.447108 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.447073 5433 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 14:58:16.448746 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.448715 5433 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 14:58:16.448746 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.448740 5433 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 14:58:16.448935 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.448762 5433 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 14:58:16.448935 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:16.448797 5433 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 14:58:16.457871 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.457839 5433 generic.go:356] "Generic (PLEG): container finished" podID="ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f" containerID="e3caca96c848414dd05b1f3ef525325c93c3b33b17dfbf27fe77d1e0cb01358f" exitCode=0 Nov 02 14:58:16.467215 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.467184 5433 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="80f80e0fc7255358158b5271c185bfe278fbe50e811658b33837de57ec7976df" exitCode=0 Nov 02 14:58:16.467215 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.467207 5433 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="70aa349b8c2612b698c21f3ec39b0d33d99e8cf6a4c234bd9c9a5827cc5af753" exitCode=0 Nov 02 14:58:16.467215 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.467214 5433 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="8a8c2c28a165485cd564902af75529b29414e3d8fb0943cbcfb8bfbc082dbaa6" exitCode=0 Nov 02 14:58:16.467215 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.467220 5433 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="b45c9b6392f790d61d85d061b1958b79183cc32cf007787f5b0859647fa3907a" exitCode=0 Nov 02 14:58:16.467215 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.467225 5433 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="8cae6207f58df68ca1ffe878cb2943b027bdda6ade3e7cb023b8e4207fb64186" exitCode=0 Nov 02 14:58:16.467540 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.467230 5433 generic.go:356] "Generic (PLEG): container finished" podID="8b6bfaad-2c57-4baa-be7d-f25715574792" containerID="76c468396751e4af18f55c5d41b3001a7d138d3c342e12bc44377b5cd1aff381" exitCode=0 Nov 02 14:58:16.493621 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.493590 5433 generic.go:356] "Generic (PLEG): container finished" podID="c78c4a25b0fa3f7dd2a10621ff614039" containerID="339924069d77259f5b31ee96dfb63037399565cff30143f35eb75d90fc4e8c29" exitCode=0 Nov 02 14:58:16.510957 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:16.510927 5433 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:58:16.517254 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.517196 5433 manager.go:324] Recovery completed Nov 02 14:58:16.538133 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.538110 5433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:58:16.540258 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.540227 5433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:58:16.540373 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.540264 5433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:58:16.540373 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.540276 5433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:58:16.543046 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543025 5433 cpu_manager.go:226] "Starting" policy="none" Nov 02 14:58:16.543046 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543038 5433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:58:16.543185 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543042 5433 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 14:58:16.543185 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543113 5433 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 14:58:16.543277 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543239 5433 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 14:58:16.543277 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543251 5433 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 14:58:16.543277 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543268 5433 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 14:58:16.543277 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543273 5433 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 14:58:16.543277 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543279 5433 policy_none.go:50] "Start" Nov 02 14:58:16.543419 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543289 5433 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 14:58:16.543419 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543298 5433 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 14:58:16.543474 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543429 5433 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 14:58:16.543474 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543436 5433 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 14:58:16.543474 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543441 5433 policy_none.go:44] "Start" Nov 02 14:58:16.543933 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543914 5433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:58:16.544008 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543950 5433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:58:16.544008 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.543966 5433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:58:16.548437 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.548409 5433 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 14:58:16.548955 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:16.548932 5433 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 14:58:16.552056 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.552043 5433 manager.go:341] "Starting Device Plugin manager" Nov 02 14:58:16.552134 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:16.552068 5433 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 14:58:16.552134 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.552077 5433 server.go:87] "Starting device plugin registration server" Nov 02 14:58:16.552398 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.552386 5433 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 14:58:16.552451 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.552402 5433 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 14:58:16.552521 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.552500 5433 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 14:58:16.552644 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.552612 5433 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 14:58:16.552709 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.552655 5433 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 14:58:16.552709 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.552663 5433 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 14:58:16.552781 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.552696 5433 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:58:16.552643264Z","UUID":"82bf8e79-fd7b-4270-aa02-6c90a75d967a","Handler":null,"Name":"","Endpoint":""} Nov 02 14:58:16.553842 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.553825 5433 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 14:58:16.553905 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.553848 5433 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 14:58:16.555938 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:16.555914 5433 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 14:58:16.556038 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:16.555965 5433 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-181.us-west-2.compute.internal\" not found" Nov 02 14:58:16.598924 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.598891 5433 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:58:16.599073 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.598966 5433 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:58:16.599073 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.599046 5433 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 14:58:16.602256 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.602238 5433 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:58:16.653031 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.653003 5433 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:58:16.654666 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.654650 5433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:58:16.654740 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.654679 5433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:58:16.654740 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.654690 5433 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-181.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:58:16.654740 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.654729 5433 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.688453 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.688427 5433 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:58:16.749482 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.749427 5433 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal"] Nov 02 14:58:16.750155 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.750109 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"c78c4a25b0fa3f7dd2a10621ff614039","Type":"ContainerStarted","Data":"6d16f7536cf5be2f51d91bc605d80ff9c791c1a35961eaa50f4a58ad4d249044"} Nov 02 14:58:16.750195 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.750172 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"c78c4a25b0fa3f7dd2a10621ff614039","Type":"ContainerDied","Data":"339924069d77259f5b31ee96dfb63037399565cff30143f35eb75d90fc4e8c29"} Nov 02 14:58:16.750195 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.750188 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"c78c4a25b0fa3f7dd2a10621ff614039","Type":"ContainerStarted","Data":"2fdade16c0d70ce01cd2a8280a9d25215113305d0f78e6caafa7938ae784b9f4"} Nov 02 14:58:16.750266 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.750201 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"010a8efd81cfe8476c37b8d73ee870b2","Type":"ContainerStarted","Data":"cdb93fdbc918aabcf8f2767408d20dd1ca82bd837a61effa00cc9826bf752bae"} Nov 02 14:58:16.750266 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.750208 5433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.750266 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.750217 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" event={"ID":"010a8efd81cfe8476c37b8d73ee870b2","Type":"ContainerStarted","Data":"a5c71fadb1c1bc9e1fc6f904ccea9974225acd46a547316a4b537794092e5c12"} Nov 02 14:58:16.750344 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.750208 5433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.769287 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.769224 5433 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.769402 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.769292 5433 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.813953 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.813921 5433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:58:16.814099 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:16.813971 5433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.814604 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.814586 5433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:58:16.814674 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:16.814638 5433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.900543 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.900510 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/010a8efd81cfe8476c37b8d73ee870b2-config\") pod \"kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"010a8efd81cfe8476c37b8d73ee870b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.900720 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.900550 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:16.900720 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:16.900631 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.001220 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.001180 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/010a8efd81cfe8476c37b8d73ee870b2-config\") pod \"kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"010a8efd81cfe8476c37b8d73ee870b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.001220 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.001130 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/010a8efd81cfe8476c37b8d73ee870b2-config\") pod \"kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"010a8efd81cfe8476c37b8d73ee870b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.001402 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.001239 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.001402 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.001258 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.001402 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.001300 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.001402 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.001310 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c78c4a25b0fa3f7dd2a10621ff614039-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" (UID: \"c78c4a25b0fa3f7dd2a10621ff614039\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.393964 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.393930 5433 apiserver.go:52] "Watching apiserver" Nov 02 14:58:17.490684 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.490641 5433 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 14:58:17.491387 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.491356 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-sx5r5","openshift-network-operator/iptables-alerter-px579","openshift-dns/dns-default-lzs9f","openshift-dns/node-resolver-2zz8m","openshift-insights/insights-runtime-extractor-4njs7","openshift-network-diagnostics/network-check-target-l6p7c","openshift-ingress-canary/ingress-canary-vr5nl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal","kube-system/konnectivity-agent-4522z","kube-system/global-pull-secret-syncer-69x5r","openshift-cluster-node-tuning-operator/tuned-kcfxh","openshift-image-registry/node-ca-jpfd7","openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv","openshift-ovn-kubernetes/ovnkube-node-mqvnb","openshift-multus/multus-frtdt","kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q","openshift-monitoring/metrics-server-74bdbddd8-wl9lp","openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b","openshift-monitoring/node-exporter-d4l7z","openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b","openshift-multus/multus-additional-cni-plugins-9j6h2"] Nov 02 14:58:17.506843 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.506816 5433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.507005 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.506944 5433 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.512864 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.512846 5433 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 14:58:17.556073 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.556042 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 14:58:17.556073 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.556072 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q62fl\"" Nov 02 14:58:17.556297 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.556175 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hnsn9\"" Nov 02 14:58:17.556297 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.556267 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tsn9p\"" Nov 02 14:58:17.571452 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.571423 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 14:58:17.571614 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.571595 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.571678 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.571663 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 14:58:17.571738 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.571685 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.571899 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.571882 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.571994 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.571931 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.572126 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572106 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.572126 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572103 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 14:58:17.572289 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572108 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 14:58:17.572289 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572151 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.572289 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572168 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.572289 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572185 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 14:58:17.572289 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572202 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.572289 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572220 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.572289 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572288 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 14:58:17.572482 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572302 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.572482 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572310 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 14:58:17.572482 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572386 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.572559 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572525 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.572749 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572734 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 14:58:17.572834 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572821 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.572873 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572863 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 14:58:17.572935 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572918 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.573002 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572936 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.573002 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572928 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.573002 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572942 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 14:58:17.573002 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.572998 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.604097 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604063 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2616b3d7-11af-4008-99e7-44a5c575e85a-konnectivity-ca\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:58:17.604248 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604100 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-modprobe-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.604248 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604121 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-kubernetes\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.604248 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604135 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-run\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.604248 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604160 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-sys\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.604248 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604201 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-cnibin\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.604410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604326 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-config-volume\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:17.604410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604368 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2616b3d7-11af-4008-99e7-44a5c575e85a-konnectivity-ca\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:58:17.604410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604381 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.604494 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604413 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dn7\" (UniqueName: \"kubernetes.io/projected/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-kube-api-access-w7dn7\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.604494 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604481 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-tuned\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.604574 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604559 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-tuned\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.604632 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604615 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-binary-copy\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.604677 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604642 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-bin\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.604677 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604658 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-dbus\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:17.604756 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604680 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7m4\" (UniqueName: \"kubernetes.io/projected/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-api-access-bn7m4\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.604756 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604698 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:17.604756 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604736 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.604924 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604773 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-binary-copy\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.604924 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604808 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2616b3d7-11af-4008-99e7-44a5c575e85a-agent-certs\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:58:17.604924 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604883 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gzz\" (UniqueName: \"kubernetes.io/projected/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-kube-api-access-96gzz\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:17.604924 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604899 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-grpc-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.604924 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604922 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-var-lib-kubelet\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.605125 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604938 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-conf-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.605125 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.604996 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-crio-socket\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.605125 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605101 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-etc-selinux\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.605222 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605160 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhzc\" (UniqueName: \"kubernetes.io/projected/80cddaf6-6206-4424-bff1-953ac57d7c81-kube-api-access-6hhzc\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.605222 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605193 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-client-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.605325 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605284 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-sys\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.605362 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605351 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-system-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.605399 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605370 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cni-binary-copy\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.605428 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605396 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-slash\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.605428 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605420 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-tls\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.605501 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605444 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.605501 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605462 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-kubelet-config\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:17.605501 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605486 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-hostroot\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.605613 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605504 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxbs\" (UniqueName: \"kubernetes.io/projected/e30a530f-5e3c-48f7-8872-f0b58d08a91e-kube-api-access-2hxbs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.605613 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605535 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.605613 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605573 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cni-binary-copy\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.605713 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605655 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9695a92-acd9-41de-bfcf-13d149ecb730-tmp-dir\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:58:17.605764 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605719 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0364fe3-6710-4de9-af3b-cd66989104f8-iptables-alerter-script\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:58:17.605810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605777 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9695a92-acd9-41de-bfcf-13d149ecb730-tmp-dir\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:58:17.605810 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605793 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlfss\" (UniqueName: \"kubernetes.io/projected/2dc3e10d-fde4-4b8a-97b7-e065848bed64-kube-api-access-xlfss\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.605901 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605819 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-var-lib-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.605901 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605841 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:58:17.605901 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605858 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-cert\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:58:17.605901 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605889 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83a2eda6-8157-428a-b0f3-c2cc77563252-metrics-client-ca\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.606167 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605951 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.606167 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.605988 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-systemd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.606167 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606032 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.606167 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606042 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a0364fe3-6710-4de9-af3b-cd66989104f8-iptables-alerter-script\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:58:17.606167 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606131 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-wtmp\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.606167 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606161 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.606364 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606181 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mr5q\" (UniqueName: \"kubernetes.io/projected/8b6bfaad-2c57-4baa-be7d-f25715574792-kube-api-access-4mr5q\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.606364 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606202 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-daemon-config\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.606364 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606232 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-registration-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.606364 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606259 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.606364 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606285 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795a4ecf-3ca0-4699-a06a-a051078f9fd1-host\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:58:17.606524 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606371 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.606524 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606412 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-system-cni-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.606524 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606433 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-bin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.606524 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606464 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-daemon-config\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.606524 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606488 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.606524 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606512 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.606697 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606526 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-root\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.606697 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606541 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-systemd\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.606697 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606579 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.606697 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606608 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:17.606697 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606662 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:17.606861 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606717 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-kubelet\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.606861 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606737 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-systemd-units\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.606861 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606756 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.606861 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606776 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0364fe3-6710-4de9-af3b-cd66989104f8-host-slash\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:58:17.606861 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606806 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-tls\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.606861 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606823 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b6bfaad-2c57-4baa-be7d-f25715574792-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.607075 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606860 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.607075 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606899 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-ovn\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.607075 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.606942 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jvst\" (UniqueName: \"kubernetes.io/projected/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-kube-api-access-6jvst\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.607075 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607005 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysconfig\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.607215 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607112 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6kz\" (UniqueName: \"kubernetes.io/projected/9ac6fea3-ce51-474e-bb44-9b328a950fbe-kube-api-access-7b6kz\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.607215 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607141 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovn-node-metrics-cert\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.607215 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607167 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-socket-dir-parent\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.607215 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607205 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.607415 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607264 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-textfile\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.607415 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607286 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-k8s-cni-cncf-io\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.607415 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607375 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-textfile\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.607415 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607381 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-socket-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.607415 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607412 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cnibin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.607620 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607439 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-config\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.607620 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607473 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9695a92-acd9-41de-bfcf-13d149ecb730-hosts-file\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:58:17.607620 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607571 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-929p7\" (UniqueName: \"kubernetes.io/projected/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-kube-api-access-929p7\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:17.607620 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607597 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-kubelet\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.607816 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607660 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-config\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.607816 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607688 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-node-log\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.607816 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607723 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-log-socket\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.607816 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607765 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.607816 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607796 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.608045 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607866 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsqxh\" (UniqueName: \"kubernetes.io/projected/e9695a92-acd9-41de-bfcf-13d149ecb730-kube-api-access-dsqxh\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:58:17.608045 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607889 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-host\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.608045 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607905 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-netns\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.608045 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607921 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-env-overrides\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.608045 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607946 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-device-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.608045 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.607964 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-lib-modules\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.608045 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608005 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-etc-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.608045 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608037 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-sys-fs\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608073 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-tmp\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608084 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-env-overrides\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608118 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608151 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtd5\" (UniqueName: \"kubernetes.io/projected/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-api-access-7jtd5\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608174 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608245 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd0ae468-b91d-4121-baca-0cc3836997ae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608263 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q976l\" (UniqueName: \"kubernetes.io/projected/a0364fe3-6710-4de9-af3b-cd66989104f8-kube-api-access-q976l\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608257 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dc3e10d-fde4-4b8a-97b7-e065848bed64-tmp\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608280 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-conf\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608296 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-multus-certs\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608312 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-metrics-server-audit-profiles\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608340 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd0ae468-b91d-4121-baca-0cc3836997ae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.608363 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608352 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.608818 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608396 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ll25\" (UniqueName: \"kubernetes.io/projected/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-kube-api-access-7ll25\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:58:17.608818 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608421 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a2eda6-8157-428a-b0f3-c2cc77563252-tmp\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.608818 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608447 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.608818 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608598 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a2eda6-8157-428a-b0f3-c2cc77563252-tmp\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.608818 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608604 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-netns\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.608818 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608645 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:17.608818 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608734 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795a4ecf-3ca0-4699-a06a-a051078f9fd1-serviceca\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:58:17.608818 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608774 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsqwp\" (UniqueName: \"kubernetes.io/projected/795a4ecf-3ca0-4699-a06a-a051078f9fd1-kube-api-access-hsqwp\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:58:17.608818 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608804 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-netd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.609180 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.608863 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.609180 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609019 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:58:17.609180 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609042 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.609180 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609062 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.609180 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609087 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e30a530f-5e3c-48f7-8872-f0b58d08a91e-audit-log\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.609180 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609112 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb969\" (UniqueName: \"kubernetes.io/projected/6c7d4401-1b82-4665-8838-9e05fac70e69-kube-api-access-nb969\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:58:17.609180 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609138 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-metrics-client-ca\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.609180 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609166 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcrc\" (UniqueName: \"kubernetes.io/projected/83a2eda6-8157-428a-b0f3-c2cc77563252-kube-api-access-5jcrc\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.609180 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609174 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e30a530f-5e3c-48f7-8872-f0b58d08a91e-audit-log\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.609468 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609195 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-multus\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.609468 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609222 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-metrics-tls\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:17.609468 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609247 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-tmp-dir\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:17.609468 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609272 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phl58\" (UniqueName: \"kubernetes.io/projected/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-kube-api-access-phl58\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:17.609468 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609297 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-etc-kubernetes\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.609468 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609333 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-tmp-dir\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:17.609468 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609374 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/795a4ecf-3ca0-4699-a06a-a051078f9fd1-serviceca\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:58:17.609468 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609381 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-script-lib\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.609468 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609408 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.609746 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609479 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-data-volume\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.609746 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609500 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-client-certs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.609746 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609580 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovnkube-script-lib\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.609746 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.609647 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-data-volume\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.610952 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.610937 5433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:58:17.611017 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:17.611002 5433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.611358 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.611339 5433 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:58:17.611462 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:17.611382 5433 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal" Nov 02 14:58:17.655192 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.655104 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s2tzl\"" Nov 02 14:58:17.655455 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.655438 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 14:58:17.655602 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.655584 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bxvmm\"" Nov 02 14:58:17.655713 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.655701 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 14:58:17.655833 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.655816 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 14:58:17.655906 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.655820 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 14:58:17.655906 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.655838 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 14:58:17.656107 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656087 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 14:58:17.656107 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656102 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 14:58:17.656252 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656093 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 14:58:17.656252 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656226 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 14:58:17.656252 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656237 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dlx4j\"" Nov 02 14:58:17.656383 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656255 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-djmct\"" Nov 02 14:58:17.656383 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656277 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 14:58:17.656383 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656343 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 14:58:17.656520 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656420 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ct4hh\"" Nov 02 14:58:17.656520 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656418 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4d7td\"" Nov 02 14:58:17.656520 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656439 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b9vbw\"" Nov 02 14:58:17.656520 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656449 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 14:58:17.656520 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656518 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:58:17.656689 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656527 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gmvcm\"" Nov 02 14:58:17.656689 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656548 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4crbl\"" Nov 02 14:58:17.656689 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656593 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-29n45\"" Nov 02 14:58:17.656689 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656636 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2tgnv\"" Nov 02 14:58:17.656689 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656647 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 14:58:17.656846 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656808 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 14:58:17.656846 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656834 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2bsvz\"" Nov 02 14:58:17.656932 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656853 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dcs5n\"" Nov 02 14:58:17.656932 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656896 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c2bv5\"" Nov 02 14:58:17.656932 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656902 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 14:58:17.656932 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656911 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 14:58:17.657137 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656932 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 14:58:17.657137 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.656943 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cr8rg\"" Nov 02 14:58:17.657137 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657007 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zw4g4\"" Nov 02 14:58:17.657137 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657007 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-67fcni9nmp7r6\"" Nov 02 14:58:17.657137 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657019 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fgvhh\"" Nov 02 14:58:17.657374 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657148 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 14:58:17.657374 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657187 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gwmlb\"" Nov 02 14:58:17.657374 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657264 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.657374 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657299 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2bv74s1v9qjn0\"" Nov 02 14:58:17.657374 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657336 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 14:58:17.657678 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657485 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-ovn-node-metrics-cert\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.657678 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657593 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.657811 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657791 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-tls\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.657884 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.657872 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:58:17.658044 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.658027 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.658415 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.658399 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:17.659546 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.659524 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.659667 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.659569 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c7d4401-1b82-4665-8838-9e05fac70e69-metrics-certs\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:58:17.659667 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.659632 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.659811 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.659694 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-metrics-tls\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:17.659811 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.659771 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed84914e-49d4-4f14-877b-ca1b0cdff144-original-pull-secret\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:17.659811 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.659785 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-client-certs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.664926 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.664908 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.665032 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.664942 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.665095 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.665082 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2616b3d7-11af-4008-99e7-44a5c575e85a-agent-certs\") pod \"konnectivity-agent-4522z\" (UID: \"2616b3d7-11af-4008-99e7-44a5c575e85a\") " pod="kube-system/konnectivity-agent-4522z" Nov 02 14:58:17.665137 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.665091 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:17.665249 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.665232 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-grpc-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.666098 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.666066 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.666187 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.666095 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-secret-metrics-server-tls\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.666187 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.666084 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e30a530f-5e3c-48f7-8872-f0b58d08a91e-client-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.666313 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.666300 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-cert\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:58:17.667153 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.667136 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:17.667211 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.667158 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83a2eda6-8157-428a-b0f3-c2cc77563252-secret-thanos-querier-tls\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.670749 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.670731 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 14:58:17.670992 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.670959 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 14:58:17.671252 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.671238 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.671283 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.671262 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.671418 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.671398 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.671464 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.671429 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 14:58:17.671464 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.671439 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 14:58:17.671603 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.671592 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 14:58:17.671636 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.671598 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 14:58:17.671911 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.671895 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 14:58:17.671911 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.671908 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 14:58:17.672105 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.672088 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 14:58:17.675171 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.675156 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-config-volume\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:17.676313 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.676295 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.676359 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.676333 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83a2eda6-8157-428a-b0f3-c2cc77563252-metrics-client-ca\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.676438 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.676423 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.677520 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.677492 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:17.677813 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.677796 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dn7\" (UniqueName: \"kubernetes.io/projected/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-kube-api-access-w7dn7\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.678685 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.678667 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd0ae468-b91d-4121-baca-0cc3836997ae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.678760 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.678697 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-metrics-server-audit-profiles\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.679871 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.679856 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-metrics-client-ca\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.679958 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.679943 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e30a530f-5e3c-48f7-8872-f0b58d08a91e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.680036 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.679999 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.680263 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.680247 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96gzz\" (UniqueName: \"kubernetes.io/projected/e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3-kube-api-access-96gzz\") pod \"openshift-state-metrics-7bd7489957-nhf9q\" (UID: \"e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-nhf9q" Nov 02 14:58:17.686442 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.686419 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtd5\" (UniqueName: \"kubernetes.io/projected/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-kube-api-access-7jtd5\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.686527 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.686493 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlfss\" (UniqueName: \"kubernetes.io/projected/2dc3e10d-fde4-4b8a-97b7-e065848bed64-kube-api-access-xlfss\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.687183 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.687167 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6kz\" (UniqueName: \"kubernetes.io/projected/9ac6fea3-ce51-474e-bb44-9b328a950fbe-kube-api-access-7b6kz\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.691614 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.691596 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7m4\" (UniqueName: \"kubernetes.io/projected/bd0ae468-b91d-4121-baca-0cc3836997ae-kube-api-access-bn7m4\") pod \"kube-state-metrics-5dc8d7d89b-jqdhk\" (UID: \"bd0ae468-b91d-4121-baca-0cc3836997ae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqdhk" Nov 02 14:58:17.692630 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.692609 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb969\" (UniqueName: \"kubernetes.io/projected/6c7d4401-1b82-4665-8838-9e05fac70e69-kube-api-access-nb969\") pod \"network-metrics-daemon-sx5r5\" (UID: \"6c7d4401-1b82-4665-8838-9e05fac70e69\") " pod="openshift-multus/network-metrics-daemon-sx5r5" Nov 02 14:58:17.692782 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.692768 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ll25\" (UniqueName: \"kubernetes.io/projected/888654bd-3ef7-4ff1-b31e-0ce4be3587b6-kube-api-access-7ll25\") pod \"ingress-canary-vr5nl\" (UID: \"888654bd-3ef7-4ff1-b31e-0ce4be3587b6\") " pod="openshift-ingress-canary/ingress-canary-vr5nl" Nov 02 14:58:17.692814 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.692793 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jvst\" (UniqueName: \"kubernetes.io/projected/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-kube-api-access-6jvst\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.693589 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.693573 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q976l\" (UniqueName: \"kubernetes.io/projected/a0364fe3-6710-4de9-af3b-cd66989104f8-kube-api-access-q976l\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:58:17.693856 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.693842 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsqwp\" (UniqueName: \"kubernetes.io/projected/795a4ecf-3ca0-4699-a06a-a051078f9fd1-kube-api-access-hsqwp\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:58:17.693948 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.693934 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phl58\" (UniqueName: \"kubernetes.io/projected/5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6-kube-api-access-phl58\") pod \"monitoring-plugin-77f4f76594-mxp9b\" (UID: \"5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:17.694705 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.694677 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-929p7\" (UniqueName: \"kubernetes.io/projected/d4f454c8-54e1-4eba-a8ea-10a32729ce0e-kube-api-access-929p7\") pod \"dns-default-lzs9f\" (UID: \"d4f454c8-54e1-4eba-a8ea-10a32729ce0e\") " pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:17.694787 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.694699 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhzc\" (UniqueName: \"kubernetes.io/projected/80cddaf6-6206-4424-bff1-953ac57d7c81-kube-api-access-6hhzc\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.696575 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.696559 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxbs\" (UniqueName: \"kubernetes.io/projected/e30a530f-5e3c-48f7-8872-f0b58d08a91e-kube-api-access-2hxbs\") pod \"metrics-server-74bdbddd8-wl9lp\" (UID: \"e30a530f-5e3c-48f7-8872-f0b58d08a91e\") " pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:17.698912 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.698896 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsqxh\" (UniqueName: \"kubernetes.io/projected/e9695a92-acd9-41de-bfcf-13d149ecb730-kube-api-access-dsqxh\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:58:17.699037 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.699021 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mr5q\" (UniqueName: \"kubernetes.io/projected/8b6bfaad-2c57-4baa-be7d-f25715574792-kube-api-access-4mr5q\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.699399 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.699380 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdzxj\" (UniqueName: \"kubernetes.io/projected/8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4-kube-api-access-gdzxj\") pod \"network-check-target-l6p7c\" (UID: \"8dc0d2d5-f127-4b55-bed1-0bbfffb31ff4\") " pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:58:17.699431 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.699387 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcrc\" (UniqueName: \"kubernetes.io/projected/83a2eda6-8157-428a-b0f3-c2cc77563252-kube-api-access-5jcrc\") pod \"thanos-querier-5b4c77bc7-qtd5b\" (UID: \"83a2eda6-8157-428a-b0f3-c2cc77563252\") " pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:17.709806 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709788 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-socket-dir-parent\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.709865 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709814 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-k8s-cni-cncf-io\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.709865 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709831 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-socket-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.709865 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709845 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cnibin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.709865 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709861 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9695a92-acd9-41de-bfcf-13d149ecb730-hosts-file\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709889 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-kubelet\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709905 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-k8s-cni-cncf-io\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709910 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-socket-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709922 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-cnibin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709914 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-node-log\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709935 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9695a92-acd9-41de-bfcf-13d149ecb730-hosts-file\") pod \"node-resolver-2zz8m\" (UID: \"e9695a92-acd9-41de-bfcf-13d149ecb730\") " pod="openshift-dns/node-resolver-2zz8m" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709947 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-socket-dir-parent\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709965 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-log-socket\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.709967 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-kubelet\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710015 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710035 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710030 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-log-socket\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710051 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-node-log\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710066 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-host\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710070 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710107 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-netns\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710143 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-host\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710170 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-device-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710200 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-netns\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710204 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-device-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710223 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-lib-modules\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710243 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-etc-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710281 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-etc-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710290 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-sys-fs\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710309 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-lib-modules\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710313 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710339 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-conf\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710377 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-multus-certs\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710396 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710410 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-run-multus-certs\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710391 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-sys-fs\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.710410 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710420 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710442 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-conf\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710440 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-netns\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710465 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-netns\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710480 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-netd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710500 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-multus\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710516 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-etc-kubernetes\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710532 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-modprobe-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710546 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-kubernetes\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710558 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysctl-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710565 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-run\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710583 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-multus\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710588 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-etc-kubernetes\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710602 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-sys\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710616 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-kubernetes\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710626 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-modprobe-d\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710618 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-run\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710645 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-netd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710649 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-cnibin\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710683 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-bin\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710685 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-cnibin\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710654 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-sys\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710761 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-dbus\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710795 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-var-lib-kubelet\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710802 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-dbus\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710765 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-cni-bin\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710822 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-conf-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710860 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-var-lib-kubelet\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710866 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-crio-socket\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710890 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-etc-selinux\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710914 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-sys\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710926 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-conf-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710936 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-system-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710949 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7df6cc-81c7-4ecc-b1f8-4b8de2464445-crio-socket\") pod \"insights-runtime-extractor-4njs7\" (UID: \"ab7df6cc-81c7-4ecc-b1f8-4b8de2464445\") " pod="openshift-insights/insights-runtime-extractor-4njs7" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710984 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-sys\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711001 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-slash\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.710944 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-etc-selinux\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711021 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-system-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711041 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-slash\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711044 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-kubelet-config\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711064 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-hostroot\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711098 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-hostroot\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711111 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed84914e-49d4-4f14-877b-ca1b0cdff144-kubelet-config\") pod \"global-pull-secret-syncer-69x5r\" (UID: \"ed84914e-49d4-4f14-877b-ca1b0cdff144\") " pod="kube-system/global-pull-secret-syncer-69x5r" Nov 02 14:58:17.711204 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711135 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-var-lib-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711259 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-var-lib-openvswitch\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711714 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711791 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-systemd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711814 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-wtmp\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711842 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-registration-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711867 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-multus-cni-dir\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711863 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795a4ecf-3ca0-4699-a06a-a051078f9fd1-host\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711901 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/795a4ecf-3ca0-4699-a06a-a051078f9fd1-host\") pod \"node-ca-jpfd7\" (UID: \"795a4ecf-3ca0-4699-a06a-a051078f9fd1\") " pod="openshift-image-registry/node-ca-jpfd7" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711925 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-systemd\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711967 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-node-exporter-wtmp\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.711955 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-registration-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712013 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-system-cni-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712109 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-system-cni-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712171 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-bin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712205 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712263 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712291 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-root\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712315 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-systemd\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712386 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-kubelet\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712414 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-systemd-units\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712462 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0364fe3-6710-4de9-af3b-cd66989104f8-host-slash\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712484 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712509 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-ovn\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712580 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysconfig\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712758 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-sysconfig\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712855 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ac6fea3-ce51-474e-bb44-9b328a950fbe-host-var-lib-cni-bin\") pod \"multus-frtdt\" (UID: \"9ac6fea3-ce51-474e-bb44-9b328a950fbe\") " pod="openshift-multus/multus-frtdt" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712913 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-run-ovn-kubernetes\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.712943 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.712961 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80cddaf6-6206-4424-bff1-953ac57d7c81-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4x6hv\" (UID: \"80cddaf6-6206-4424-bff1-953ac57d7c81\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4x6hv" Nov 02 14:58:17.714265 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.713045 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f-root\") pod \"node-exporter-d4l7z\" (UID: \"ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f\") " pod="openshift-monitoring/node-exporter-d4l7z" Nov 02 14:58:17.714265 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.713077 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-systemd-units\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.714265 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.713109 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0364fe3-6710-4de9-af3b-cd66989104f8-host-slash\") pod \"iptables-alerter-px579\" (UID: \"a0364fe3-6710-4de9-af3b-cd66989104f8\") " pod="openshift-network-operator/iptables-alerter-px579" Nov 02 14:58:17.714265 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.713151 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2dc3e10d-fde4-4b8a-97b7-e065848bed64-etc-systemd\") pod \"tuned-kcfxh\" (UID: \"2dc3e10d-fde4-4b8a-97b7-e065848bed64\") " pod="openshift-cluster-node-tuning-operator/tuned-kcfxh" Nov 02 14:58:17.714265 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.713189 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b6bfaad-2c57-4baa-be7d-f25715574792-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9j6h2\" (UID: \"8b6bfaad-2c57-4baa-be7d-f25715574792\") " pod="openshift-multus/multus-additional-cni-plugins-9j6h2" Nov 02 14:58:17.714265 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.713225 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-host-kubelet\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.714265 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.713263 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eff5e30-0f92-4736-a63f-c2a02ceb5a47-run-ovn\") pod \"ovnkube-node-mqvnb\" (UID: \"7eff5e30-0f92-4736-a63f-c2a02ceb5a47\") " pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:17.988956 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:17.988877 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:18.154920 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:18.154886 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:19.081526 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:19.081498 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lzs9f" Nov 02 14:58:19.208607 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:19.208568 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:19.361339 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:19.361261 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:21.487876 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:21.487846 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b4c77bc7-qtd5b" Nov 02 14:58:22.635771 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:22.635734 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:22.788691 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:22.788658 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:24.004030 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:24.003954 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:26.667144 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:26.667051 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:26.671460 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:26.671431 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mxp9b" Nov 02 14:58:26.926711 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:26.926632 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:27.239760 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:27.239669 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:58:27.774409 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:27.774384 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:27.774778 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:27.774573 5433 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:58:29.372222 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:29.372184 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:29.520543 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:29.520509 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:30.446800 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:30.446765 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:30.594500 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:30.594464 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:31.094090 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:31.094060 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:31.898497 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:31.898448 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:32.044448 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:32.044412 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:35.440268 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:35.440232 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:35.853098 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:35.853018 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:36.005919 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:36.005887 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:41.009432 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:58:41.009392 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 14:58:42.980690 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:42.980658 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mqvnb" Nov 02 14:58:45.106544 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:45.106508 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:45.232728 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:45.232694 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4522z" Nov 02 14:58:45.255185 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:45.255154 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:45.437418 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:45.437387 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:45.600895 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:45.600861 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:58:47.243112 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:47.243079 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l6p7c" Nov 02 14:58:51.098924 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:51.098895 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:58:51.102653 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:58:51.102634 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-74bdbddd8-wl9lp" Nov 02 14:59:01.890262 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:59:01.890224 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:11.010181 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:59:11.010138 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 14:59:14.808289 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:59:14.808247 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:14.959212 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:59:14.959172 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:16.191901 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:59:16.191867 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:16.341706 ip-10-0-135-181 kubenswrapper[5433]: I1102 14:59:16.341671 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 14:59:41.009538 ip-10-0-135-181 kubenswrapper[5433]: E1102 14:59:41.009498 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:00:11.009811 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:00:11.009765 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:00:13.815782 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:00:13.815740 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:13.965671 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:00:13.965623 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:16.517568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:00:16.517539 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:41.009898 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:00:41.009855 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:00:47.870334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:00:47.870284 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:00:48.022072 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:00:48.022038 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:11.009670 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:01:11.009632 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:01:16.850349 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:01:16.850310 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:38.067479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:01:38.067400 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:01:41.010282 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:01:41.010246 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:02:11.010379 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:02:11.010338 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:02:17.199164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:02:17.199132 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:17.345847 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:02:17.345810 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:30.442037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:02:30.441997 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:30.590587 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:02:30.590553 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:02:41.010810 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:02:41.010773 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:03:11.010003 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:03:11.009940 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:03:17.516561 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:03:17.516529 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:36.011113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:03:36.011079 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:03:41.010234 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:03:41.010193 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:04:11.010333 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:04:11.010296 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:04:17.836118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:17.836082 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:20.997466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:20.997430 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:21.158566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:21.158527 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:21.317862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:21.317786 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:22.120416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.120383 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq"] Nov 02 15:04:22.123131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.123112 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.203650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.203608 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.203650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.203654 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.203909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.203684 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27sb\" (UniqueName: \"kubernetes.io/projected/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-kube-api-access-d27sb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.220067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.220037 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 15:04:22.220223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.220088 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-4bxn5\"" Nov 02 15:04:22.220223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.220096 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 15:04:22.220695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.220677 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq"] Nov 02 15:04:22.304306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.304273 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.304486 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.304345 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.304486 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.304383 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27sb\" (UniqueName: \"kubernetes.io/projected/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-kube-api-access-d27sb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.304849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.304820 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.304934 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.304827 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.359592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.359562 5433 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 15:04:22.361728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.361701 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27sb\" (UniqueName: \"kubernetes.io/projected/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-kube-api-access-d27sb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.436542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.436516 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:22.590738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.590707 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww"] Nov 02 15:04:22.593270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.593246 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.616322 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.616293 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:22.644103 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.644079 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 15:04:22.645099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.645083 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 15:04:22.645162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.645089 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 15:04:22.645896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.645882 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 15:04:22.645957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.645884 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 15:04:22.645957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.645910 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 15:04:22.646052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.645994 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wfw99\"" Nov 02 15:04:22.666278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.666246 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww"] Nov 02 15:04:22.707509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.707425 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.707509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.707467 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jtj\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-kube-api-access-f5jtj\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.707678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.707564 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.707678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.707618 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.707678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.707656 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.707807 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.707681 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef6be-a6dc-443d-91f2-ab4a863b8779-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.707807 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.707773 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.778244 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.778207 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:22.808282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.808251 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jtj\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-kube-api-access-f5jtj\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.808420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.808290 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.808420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.808316 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.808420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.808349 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.808420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.808385 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef6be-a6dc-443d-91f2-ab4a863b8779-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.808653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.808421 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.808653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.808457 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.809071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.809044 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.810841 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.810814 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef6be-a6dc-443d-91f2-ab4a863b8779-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.810923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.810842 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.811190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.811173 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.811268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.811247 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.827891 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.827865 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq"] Nov 02 15:04:22.831245 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:04:22.831220 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95bbc87_41e4_412c_8229_0b3aaa6bb9a1.slice/crio-368c834b9ffd118e0c0c46f8c58c9e4c6e6aa9ada6fa981ac8c1c4da35df823b WatchSource:0}: Error finding container 368c834b9ffd118e0c0c46f8c58c9e4c6e6aa9ada6fa981ac8c1c4da35df823b: Status 404 returned error can't find the container with id 368c834b9ffd118e0c0c46f8c58c9e4c6e6aa9ada6fa981ac8c1c4da35df823b Nov 02 15:04:22.833261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.833246 5433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:04:22.866700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.866662 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.866809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.866731 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jtj\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-kube-api-access-f5jtj\") pod \"istiod-openshift-gateway-7876bfcb74-jzdww\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:22.903474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:22.903449 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:23.318189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:23.318155 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww"] Nov 02 15:04:23.321243 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:04:23.321206 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cef6be_a6dc_443d_91f2_ab4a863b8779.slice/crio-18a6c6bd9cd83b6290d1cecd3812d0888d9d5483e6cc8423e644b86fc71c622f WatchSource:0}: Error finding container 18a6c6bd9cd83b6290d1cecd3812d0888d9d5483e6cc8423e644b86fc71c622f: Status 404 returned error can't find the container with id 18a6c6bd9cd83b6290d1cecd3812d0888d9d5483e6cc8423e644b86fc71c622f Nov 02 15:04:23.696170 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:23.696134 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" event={"ID":"20cef6be-a6dc-443d-91f2-ab4a863b8779","Type":"ContainerStarted","Data":"18a6c6bd9cd83b6290d1cecd3812d0888d9d5483e6cc8423e644b86fc71c622f"} Nov 02 15:04:23.697191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:23.697170 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" event={"ID":"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1","Type":"ContainerStarted","Data":"368c834b9ffd118e0c0c46f8c58c9e4c6e6aa9ada6fa981ac8c1c4da35df823b"} Nov 02 15:04:24.119511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:24.119425 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:28.760320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:28.760282 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:04:28.760669 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:28.760366 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:04:29.722804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:29.722759 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" event={"ID":"20cef6be-a6dc-443d-91f2-ab4a863b8779","Type":"ContainerStarted","Data":"d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e"} Nov 02 15:04:29.722997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:29.722865 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:29.724908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:29.724874 5433 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-jzdww container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:04:29.725048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:29.724933 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" podUID="20cef6be-a6dc-443d-91f2-ab4a863b8779" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:04:29.922203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:29.922160 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" podStartSLOduration=2.486322284 podStartE2EDuration="7.922146648s" totalImagesPullingTime="5.435824364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:29.883199551 +0000 UTC m=+373.556402742" watchObservedRunningTime="2026-11-02 15:04:29.922146648 +0000 UTC m=+373.595349842" Nov 02 15:04:30.272824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:30.272789 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:30.431357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:30.431326 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:30.727080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:30.727052 5433 generic.go:356] "Generic (PLEG): container finished" podID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerID="03419b09c6aa94b6da31a4949309c0377aa10a81b6d43f37ee3230c89b1e28b1" exitCode=0 Nov 02 15:04:30.727243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:30.727160 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" event={"ID":"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1","Type":"ContainerDied","Data":"03419b09c6aa94b6da31a4949309c0377aa10a81b6d43f37ee3230c89b1e28b1"} Nov 02 15:04:30.728029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:30.728009 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:04:30.821861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:30.821834 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:31.020191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:31.020119 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:31.165167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:31.165135 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:34.743997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:34.743937 5433 generic.go:356] "Generic (PLEG): container finished" podID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerID="d5f27efd1a9125662df74e43b902c81dd19eacda410d3bf76fd8d074f09ad617" exitCode=0 Nov 02 15:04:34.744396 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:34.744008 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" event={"ID":"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1","Type":"ContainerDied","Data":"d5f27efd1a9125662df74e43b902c81dd19eacda410d3bf76fd8d074f09ad617"} Nov 02 15:04:34.949356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:34.949328 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:35.091285 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:35.091248 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:41.013198 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:04:41.013154 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:04:42.774754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:42.774720 5433 generic.go:356] "Generic (PLEG): container finished" podID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerID="01f95f09a3ac4c5ba3b3e781375f66391d641eff34ac0cd87cc01ce1a3e4aa28" exitCode=0 Nov 02 15:04:42.775207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:42.774767 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" event={"ID":"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1","Type":"ContainerDied","Data":"01f95f09a3ac4c5ba3b3e781375f66391d641eff34ac0cd87cc01ce1a3e4aa28"} Nov 02 15:04:43.956559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:43.956518 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:44.088897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.088827 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-util\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-util\") pod \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " Nov 02 15:04:44.088897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.088895 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-bundle\") pod \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " Nov 02 15:04:44.089098 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.088927 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-kube-api-access-d27sb\" (UniqueName: \"kubernetes.io/projected/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-kube-api-access-d27sb\") pod \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\" (UID: \"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1\") " Nov 02 15:04:44.089276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.089249 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-bundle" pod "b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" (UID: "b95bbc87-41e4-412c-8229-0b3aaa6bb9a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:44.091324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.091296 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-kube-api-access-d27sb" pod "b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" (UID: "b95bbc87-41e4-412c-8229-0b3aaa6bb9a1"). InnerVolumeSpecName "kube-api-access-d27sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:04:44.093220 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.093201 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-util" pod "b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" (UID: "b95bbc87-41e4-412c-8229-0b3aaa6bb9a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:44.189804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.189769 5433 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-util\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:44.189804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.189799 5433 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-bundle\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:44.189804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.189812 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d27sb\" (UniqueName: \"kubernetes.io/projected/b95bbc87-41e4-412c-8229-0b3aaa6bb9a1-kube-api-access-d27sb\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:44.783873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.783782 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" event={"ID":"b95bbc87-41e4-412c-8229-0b3aaa6bb9a1","Type":"ContainerDied","Data":"368c834b9ffd118e0c0c46f8c58c9e4c6e6aa9ada6fa981ac8c1c4da35df823b"} Nov 02 15:04:44.783873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.783806 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n7sdq" Nov 02 15:04:44.783873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:44.783834 5433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="368c834b9ffd118e0c0c46f8c58c9e4c6e6aa9ada6fa981ac8c1c4da35df823b" Nov 02 15:04:45.344938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:45.344907 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:45.490290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:45.490260 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:58.887436 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:58.887395 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:04:59.069910 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:04:59.069860 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:00.590383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:00.590350 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:07.136502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:07.136437 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:07.285770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:07.285729 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:08.252912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.252853 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:08.592326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.592244 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8"] Nov 02 15:05:08.592621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.592602 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerName="extract" Nov 02 15:05:08.592705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.592623 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerName="extract" Nov 02 15:05:08.592705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.592668 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerName="util" Nov 02 15:05:08.592705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.592677 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerName="util" Nov 02 15:05:08.592705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.592689 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerName="pull" Nov 02 15:05:08.592705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.592697 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerName="pull" Nov 02 15:05:08.592938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.592773 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95bbc87-41e4-412c-8229-0b3aaa6bb9a1" containerName="extract" Nov 02 15:05:08.595036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.595013 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.671383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.671349 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f29a5450-9be7-49c0-94d7-9529cb6095b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.671383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.671391 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f29a5450-9be7-49c0-94d7-9529cb6095b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.671624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.671422 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddjrx\" (UniqueName: \"kubernetes.io/projected/f29a5450-9be7-49c0-94d7-9529cb6095b0-kube-api-access-ddjrx\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.671624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.671561 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f29a5450-9be7-49c0-94d7-9529cb6095b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.671624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.671615 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f29a5450-9be7-49c0-94d7-9529cb6095b0-tmp\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.671768 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.671681 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f29a5450-9be7-49c0-94d7-9529cb6095b0-manifests\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.695287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.695251 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 15:05:08.695287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.695273 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 15:05:08.695498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.695301 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bfjvh\"" Nov 02 15:05:08.695498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.695347 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 15:05:08.695498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.695350 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 15:05:08.695607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.695495 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 15:05:08.696086 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.696070 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8"] Nov 02 15:05:08.706387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.706362 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:08.772183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.772149 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddjrx\" (UniqueName: \"kubernetes.io/projected/f29a5450-9be7-49c0-94d7-9529cb6095b0-kube-api-access-ddjrx\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.772352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.772194 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f29a5450-9be7-49c0-94d7-9529cb6095b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.772352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.772229 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f29a5450-9be7-49c0-94d7-9529cb6095b0-tmp\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.772352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.772246 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f29a5450-9be7-49c0-94d7-9529cb6095b0-manifests\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.772352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.772280 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f29a5450-9be7-49c0-94d7-9529cb6095b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.772352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.772297 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f29a5450-9be7-49c0-94d7-9529cb6095b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.772762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.772731 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f29a5450-9be7-49c0-94d7-9529cb6095b0-manifests\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.774668 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.774641 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f29a5450-9be7-49c0-94d7-9529cb6095b0-tmp\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.774805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.774781 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f29a5450-9be7-49c0-94d7-9529cb6095b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.774870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.774834 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f29a5450-9be7-49c0-94d7-9529cb6095b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.774923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.774907 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f29a5450-9be7-49c0-94d7-9529cb6095b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.845802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.845731 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddjrx\" (UniqueName: \"kubernetes.io/projected/f29a5450-9be7-49c0-94d7-9529cb6095b0-kube-api-access-ddjrx\") pod \"opendatahub-operator-controller-manager-64686958df-smzq8\" (UID: \"f29a5450-9be7-49c0-94d7-9529cb6095b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:08.897053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.897021 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:08.908996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:08.908962 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:09.051766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:09.051725 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:09.421792 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:09.421751 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8"] Nov 02 15:05:09.440015 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:05:09.439986 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29a5450_9be7_49c0_94d7_9529cb6095b0.slice/crio-0f0111e527fd9a67d425dd8521b014936ec271f55f466d7be754f0f0c4fe069e WatchSource:0}: Error finding container 0f0111e527fd9a67d425dd8521b014936ec271f55f466d7be754f0f0c4fe069e: Status 404 returned error can't find the container with id 0f0111e527fd9a67d425dd8521b014936ec271f55f466d7be754f0f0c4fe069e Nov 02 15:05:09.540303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:09.540256 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:09.865032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:09.864943 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" event={"ID":"f29a5450-9be7-49c0-94d7-9529cb6095b0","Type":"ContainerStarted","Data":"0f0111e527fd9a67d425dd8521b014936ec271f55f466d7be754f0f0c4fe069e"} Nov 02 15:05:11.022594 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:05:11.022524 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:05:11.772314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:11.772271 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:12.028185 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:12.028100 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:12.203749 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:12.203707 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:13.436298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:13.436211 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:13.884205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:13.884170 5433 generic.go:356] "Generic (PLEG): container finished" podID="f29a5450-9be7-49c0-94d7-9529cb6095b0" containerID="abbdf8b2375734fa6e1b3982d2d302c81c1d874b69e5248d6602ac4e55a7b861" exitCode=0 Nov 02 15:05:13.884413 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:13.884244 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" event={"ID":"f29a5450-9be7-49c0-94d7-9529cb6095b0","Type":"ContainerDied","Data":"abbdf8b2375734fa6e1b3982d2d302c81c1d874b69e5248d6602ac4e55a7b861"} Nov 02 15:05:14.772038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:14.771999 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:14.984902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:14.984865 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:15.894106 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:15.894068 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" event={"ID":"f29a5450-9be7-49c0-94d7-9529cb6095b0","Type":"ContainerStarted","Data":"a046794b9253c58b58f241a30a7135cc7c9949334fef588cd1c971c144f94dd3"} Nov 02 15:05:15.894551 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:15.894188 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:16.191297 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:16.191195 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" podStartSLOduration=2.965182151 podStartE2EDuration="8.191174638s" totalImagesPullingTime="4.820055906s" totalInitContainerRuntime="405.936581ms" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:16.14500035 +0000 UTC m=+419.818203558" watchObservedRunningTime="2026-11-02 15:05:16.191174638 +0000 UTC m=+419.864377833" Nov 02 15:05:16.725790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:16.725725 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:16.874653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:16.874618 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:17.993599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:17.993548 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:18.192118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:18.192084 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:18.375855 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:18.375796 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:20.406206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:20.406164 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:20.906997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:20.906938 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:21.303724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:21.303628 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:22.359543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:22.359509 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:24.923952 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:24.923915 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:25.095361 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:25.095331 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:25.678320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:25.678269 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:25.841426 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:25.841391 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:26.718391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:26.718355 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:26.902828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:26.902794 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-smzq8" Nov 02 15:05:27.302634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:27.302592 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:28.753028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:28.752995 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:30.420447 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:30.420420 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:30.609623 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:30.609584 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:30.768861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:30.768785 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:31.604017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:31.603962 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:32.099825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:32.099790 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:32.363882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:32.363790 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:34.766966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:34.766926 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:36.715691 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:36.715655 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:37.345954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:37.345911 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:38.784471 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:38.784435 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-wdkh7"] Nov 02 15:05:38.787337 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:38.787313 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wdkh7" Nov 02 15:05:38.837574 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:38.837543 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 15:05:38.883848 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:38.883814 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 15:05:38.883848 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:38.883821 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xxjd6\"" Nov 02 15:05:38.884598 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:38.884577 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wdkh7"] Nov 02 15:05:38.925355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:38.925319 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ff6bdb-64ec-4422-96af-1ff4911c2cab-bound-sa-token\") pod \"cert-manager-597b95cb84-wdkh7\" (UID: \"57ff6bdb-64ec-4422-96af-1ff4911c2cab\") " pod="cert-manager/cert-manager-597b95cb84-wdkh7" Nov 02 15:05:38.925517 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:38.925368 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqnqj\" (UniqueName: \"kubernetes.io/projected/57ff6bdb-64ec-4422-96af-1ff4911c2cab-kube-api-access-jqnqj\") pod \"cert-manager-597b95cb84-wdkh7\" (UID: \"57ff6bdb-64ec-4422-96af-1ff4911c2cab\") " pod="cert-manager/cert-manager-597b95cb84-wdkh7" Nov 02 15:05:39.025943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:39.025905 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ff6bdb-64ec-4422-96af-1ff4911c2cab-bound-sa-token\") pod \"cert-manager-597b95cb84-wdkh7\" (UID: \"57ff6bdb-64ec-4422-96af-1ff4911c2cab\") " pod="cert-manager/cert-manager-597b95cb84-wdkh7" Nov 02 15:05:39.026134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:39.025954 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqnqj\" (UniqueName: \"kubernetes.io/projected/57ff6bdb-64ec-4422-96af-1ff4911c2cab-kube-api-access-jqnqj\") pod \"cert-manager-597b95cb84-wdkh7\" (UID: \"57ff6bdb-64ec-4422-96af-1ff4911c2cab\") " pod="cert-manager/cert-manager-597b95cb84-wdkh7" Nov 02 15:05:39.087406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:39.087333 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ff6bdb-64ec-4422-96af-1ff4911c2cab-bound-sa-token\") pod \"cert-manager-597b95cb84-wdkh7\" (UID: \"57ff6bdb-64ec-4422-96af-1ff4911c2cab\") " pod="cert-manager/cert-manager-597b95cb84-wdkh7" Nov 02 15:05:39.087531 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:39.087420 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqnqj\" (UniqueName: \"kubernetes.io/projected/57ff6bdb-64ec-4422-96af-1ff4911c2cab-kube-api-access-jqnqj\") pod \"cert-manager-597b95cb84-wdkh7\" (UID: \"57ff6bdb-64ec-4422-96af-1ff4911c2cab\") " pod="cert-manager/cert-manager-597b95cb84-wdkh7" Nov 02 15:05:39.112237 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:39.112205 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wdkh7" Nov 02 15:05:39.471829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:39.471796 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wdkh7"] Nov 02 15:05:39.475533 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:05:39.475507 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57ff6bdb_64ec_4422_96af_1ff4911c2cab.slice/crio-91451019241bc890ace625432d01892085132ca07e30f4099abcecc37b023171 WatchSource:0}: Error finding container 91451019241bc890ace625432d01892085132ca07e30f4099abcecc37b023171: Status 404 returned error can't find the container with id 91451019241bc890ace625432d01892085132ca07e30f4099abcecc37b023171 Nov 02 15:05:39.705027 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:39.704969 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:39.848462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:39.848385 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:39.977809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:39.977778 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wdkh7" event={"ID":"57ff6bdb-64ec-4422-96af-1ff4911c2cab","Type":"ContainerStarted","Data":"91451019241bc890ace625432d01892085132ca07e30f4099abcecc37b023171"} Nov 02 15:05:41.010470 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:05:41.010432 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:05:41.492329 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:41.492294 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:41.637108 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:41.637020 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:42.538931 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:42.538897 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:42.678128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:42.678093 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:43.076569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:43.076506 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:44.338391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.338357 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:44.482821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.482745 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:44.699887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.699843 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:44.726301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.726262 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-848b57c455-xx8nq"] Nov 02 15:05:44.729593 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.729567 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:44.836172 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.836142 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 15:05:44.836338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.836220 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 15:05:44.836338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.836304 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-plrs8\"" Nov 02 15:05:44.836923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.836898 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-848b57c455-xx8nq"] Nov 02 15:05:44.871311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.871283 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:44.875434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.875410 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15ebac82-cb6a-4535-bc22-a82bca0251d8-tmp\") pod \"kube-auth-proxy-848b57c455-xx8nq\" (UID: \"15ebac82-cb6a-4535-bc22-a82bca0251d8\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:44.875568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.875550 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ebac82-cb6a-4535-bc22-a82bca0251d8-tls-certs\") pod \"kube-auth-proxy-848b57c455-xx8nq\" (UID: \"15ebac82-cb6a-4535-bc22-a82bca0251d8\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:44.875610 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.875580 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvtkx\" (UniqueName: \"kubernetes.io/projected/15ebac82-cb6a-4535-bc22-a82bca0251d8-kube-api-access-pvtkx\") pod \"kube-auth-proxy-848b57c455-xx8nq\" (UID: \"15ebac82-cb6a-4535-bc22-a82bca0251d8\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:44.976412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.976375 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ebac82-cb6a-4535-bc22-a82bca0251d8-tls-certs\") pod \"kube-auth-proxy-848b57c455-xx8nq\" (UID: \"15ebac82-cb6a-4535-bc22-a82bca0251d8\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:44.976412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.976413 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvtkx\" (UniqueName: \"kubernetes.io/projected/15ebac82-cb6a-4535-bc22-a82bca0251d8-kube-api-access-pvtkx\") pod \"kube-auth-proxy-848b57c455-xx8nq\" (UID: \"15ebac82-cb6a-4535-bc22-a82bca0251d8\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:44.976651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.976433 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15ebac82-cb6a-4535-bc22-a82bca0251d8-tmp\") pod \"kube-auth-proxy-848b57c455-xx8nq\" (UID: \"15ebac82-cb6a-4535-bc22-a82bca0251d8\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:44.978756 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.978723 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15ebac82-cb6a-4535-bc22-a82bca0251d8-tmp\") pod \"kube-auth-proxy-848b57c455-xx8nq\" (UID: \"15ebac82-cb6a-4535-bc22-a82bca0251d8\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:44.979051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:44.979029 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ebac82-cb6a-4535-bc22-a82bca0251d8-tls-certs\") pod \"kube-auth-proxy-848b57c455-xx8nq\" (UID: \"15ebac82-cb6a-4535-bc22-a82bca0251d8\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:45.000469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:45.000332 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wdkh7" event={"ID":"57ff6bdb-64ec-4422-96af-1ff4911c2cab","Type":"ContainerStarted","Data":"be4bc315ad3865b5fb691e74ee4c2f715ea273e462f3e3c5afa234912fec5e07"} Nov 02 15:05:45.035515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:45.035487 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvtkx\" (UniqueName: \"kubernetes.io/projected/15ebac82-cb6a-4535-bc22-a82bca0251d8-kube-api-access-pvtkx\") pod \"kube-auth-proxy-848b57c455-xx8nq\" (UID: \"15ebac82-cb6a-4535-bc22-a82bca0251d8\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:45.042263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:45.042239 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" Nov 02 15:05:45.061486 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:45.061457 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:45.112606 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:45.111566 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-wdkh7" podStartSLOduration=2.398680114 podStartE2EDuration="7.111537834s" totalImagesPullingTime="4.71285772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:45.111213749 +0000 UTC m=+448.784416932" watchObservedRunningTime="2026-11-02 15:05:45.111537834 +0000 UTC m=+448.784741026" Nov 02 15:05:45.315183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:45.315094 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:45.404729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:45.404691 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-848b57c455-xx8nq"] Nov 02 15:05:45.483320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:45.483284 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:46.004607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:46.004568 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" event={"ID":"15ebac82-cb6a-4535-bc22-a82bca0251d8","Type":"ContainerStarted","Data":"c949c13f91a6859e47e41069247481e65c29cc8c3e91b529ed58ed5de6c257aa"} Nov 02 15:05:46.495033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:46.495001 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:46.635878 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:46.635847 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:46.778289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:46.778202 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:49.017562 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:49.017518 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" event={"ID":"15ebac82-cb6a-4535-bc22-a82bca0251d8","Type":"ContainerStarted","Data":"b75a11a37590d2974d2d5a9144f9588b7d627307a9eee5e750dc5e0c1ff0384a"} Nov 02 15:05:49.130956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:49.130874 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-848b57c455-xx8nq" podStartSLOduration=2.006934356 podStartE2EDuration="5.130856606s" totalImagesPullingTime="3.12392225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:49.129509566 +0000 UTC m=+452.802712761" watchObservedRunningTime="2026-11-02 15:05:49.130856606 +0000 UTC m=+452.804059801" Nov 02 15:05:49.446729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:49.446681 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:49.592599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:49.592566 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:52.676861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:52.676822 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:56.390582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:56.390505 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:56.555831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:56.555796 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:58.983038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:58.983005 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:05:59.131022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:05:59.130971 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:01.177190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.177156 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:01.264101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.264060 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f56cq"] Nov 02 15:06:01.267468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.267445 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:01.317388 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.317342 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 15:06:01.363813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.363788 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-68qgd\"" Nov 02 15:06:01.363946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.363815 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 15:06:01.363946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.363874 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 15:06:01.364613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.364596 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f56cq"] Nov 02 15:06:01.414672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.414647 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmzz\" (UniqueName: \"kubernetes.io/projected/117ef06d-ce68-4016-a18a-88db8b84cff0-kube-api-access-klmzz\") pod \"limitador-limitador-74dfcfb894-f56cq\" (UID: \"117ef06d-ce68-4016-a18a-88db8b84cff0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:01.414791 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.414680 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/117ef06d-ce68-4016-a18a-88db8b84cff0-config-file\") pod \"limitador-limitador-74dfcfb894-f56cq\" (UID: \"117ef06d-ce68-4016-a18a-88db8b84cff0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:01.488594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.488506 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:01.515180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.515154 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmzz\" (UniqueName: \"kubernetes.io/projected/117ef06d-ce68-4016-a18a-88db8b84cff0-kube-api-access-klmzz\") pod \"limitador-limitador-74dfcfb894-f56cq\" (UID: \"117ef06d-ce68-4016-a18a-88db8b84cff0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:01.515340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.515198 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/117ef06d-ce68-4016-a18a-88db8b84cff0-config-file\") pod \"limitador-limitador-74dfcfb894-f56cq\" (UID: \"117ef06d-ce68-4016-a18a-88db8b84cff0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:01.515873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.515848 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/117ef06d-ce68-4016-a18a-88db8b84cff0-config-file\") pod \"limitador-limitador-74dfcfb894-f56cq\" (UID: \"117ef06d-ce68-4016-a18a-88db8b84cff0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:01.519418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.519391 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f56cq"] Nov 02 15:06:01.576762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.576726 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmzz\" (UniqueName: \"kubernetes.io/projected/117ef06d-ce68-4016-a18a-88db8b84cff0-kube-api-access-klmzz\") pod \"limitador-limitador-74dfcfb894-f56cq\" (UID: \"117ef06d-ce68-4016-a18a-88db8b84cff0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:01.579408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.579383 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:01.638603 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.631887 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:01.943523 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:01.943394 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f56cq"] Nov 02 15:06:01.946681 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:06:01.946653 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117ef06d_ce68_4016_a18a_88db8b84cff0.slice/crio-0c4089d7a627f4141f3d240a7b99081b02dcf4e5a31b3670963afc8baeda93cf WatchSource:0}: Error finding container 0c4089d7a627f4141f3d240a7b99081b02dcf4e5a31b3670963afc8baeda93cf: Status 404 returned error can't find the container with id 0c4089d7a627f4141f3d240a7b99081b02dcf4e5a31b3670963afc8baeda93cf Nov 02 15:06:02.064805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.064770 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" event={"ID":"117ef06d-ce68-4016-a18a-88db8b84cff0","Type":"ContainerStarted","Data":"0c4089d7a627f4141f3d240a7b99081b02dcf4e5a31b3670963afc8baeda93cf"} Nov 02 15:06:02.464093 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.464038 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hngzt"] Nov 02 15:06:02.468544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.468523 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hngzt" Nov 02 15:06:02.517465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.517438 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5vb2v\"" Nov 02 15:06:02.521821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.521786 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hngzt"] Nov 02 15:06:02.624190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.624155 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrxt\" (UniqueName: \"kubernetes.io/projected/9661c8fb-1125-4c72-98f5-23f3c16bdebe-kube-api-access-tmrxt\") pod \"authorino-57f7f8659f-hngzt\" (UID: \"9661c8fb-1125-4c72-98f5-23f3c16bdebe\") " pod="kuadrant-system/authorino-57f7f8659f-hngzt" Nov 02 15:06:02.630766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.630735 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:02.724824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.724735 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrxt\" (UniqueName: \"kubernetes.io/projected/9661c8fb-1125-4c72-98f5-23f3c16bdebe-kube-api-access-tmrxt\") pod \"authorino-57f7f8659f-hngzt\" (UID: \"9661c8fb-1125-4c72-98f5-23f3c16bdebe\") " pod="kuadrant-system/authorino-57f7f8659f-hngzt" Nov 02 15:06:02.782799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.782760 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrxt\" (UniqueName: \"kubernetes.io/projected/9661c8fb-1125-4c72-98f5-23f3c16bdebe-kube-api-access-tmrxt\") pod \"authorino-57f7f8659f-hngzt\" (UID: \"9661c8fb-1125-4c72-98f5-23f3c16bdebe\") " pod="kuadrant-system/authorino-57f7f8659f-hngzt" Nov 02 15:06:02.792861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:02.792825 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:03.079590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:03.079500 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hngzt" Nov 02 15:06:03.439573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:03.439491 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hngzt"] Nov 02 15:06:03.442699 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:06:03.442658 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9661c8fb_1125_4c72_98f5_23f3c16bdebe.slice/crio-4522aaf5d80910bd4de534f3315e6ebb46fd59c867e85d68424909a33ff77f46 WatchSource:0}: Error finding container 4522aaf5d80910bd4de534f3315e6ebb46fd59c867e85d68424909a33ff77f46: Status 404 returned error can't find the container with id 4522aaf5d80910bd4de534f3315e6ebb46fd59c867e85d68424909a33ff77f46 Nov 02 15:06:04.073886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:04.073851 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hngzt" event={"ID":"9661c8fb-1125-4c72-98f5-23f3c16bdebe","Type":"ContainerStarted","Data":"4522aaf5d80910bd4de534f3315e6ebb46fd59c867e85d68424909a33ff77f46"} Nov 02 15:06:04.364332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:04.364254 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:04.505833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:04.505801 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:04.767148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:04.767105 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:04.906399 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:04.906354 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:07.089092 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:07.089059 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" event={"ID":"117ef06d-ce68-4016-a18a-88db8b84cff0","Type":"ContainerStarted","Data":"a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954"} Nov 02 15:06:07.089548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:07.089162 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:07.090559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:07.090536 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hngzt" event={"ID":"9661c8fb-1125-4c72-98f5-23f3c16bdebe","Type":"ContainerStarted","Data":"354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823"} Nov 02 15:06:07.198313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:07.198255 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" podStartSLOduration=1.900982457 podStartE2EDuration="6.198236569s" totalImagesPullingTime="4.297254112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:07.197414911 +0000 UTC m=+470.870618113" watchObservedRunningTime="2026-11-02 15:06:07.198236569 +0000 UTC m=+470.871439765" Nov 02 15:06:07.306088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:07.306035 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-hngzt" podStartSLOduration=1.890998847 podStartE2EDuration="5.306020827s" totalImagesPullingTime="3.41502198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:07.305776886 +0000 UTC m=+470.978980082" watchObservedRunningTime="2026-11-02 15:06:07.306020827 +0000 UTC m=+470.979224022" Nov 02 15:06:09.025157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:09.025117 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:09.167875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:09.167840 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:10.119357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:10.119321 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hngzt"] Nov 02 15:06:10.119720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:10.119532 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-hngzt" podUID="9661c8fb-1125-4c72-98f5-23f3c16bdebe" containerName="authorino" containerID="cri-o://354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823" gracePeriod=30 Nov 02 15:06:10.356601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:10.356579 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hngzt" Nov 02 15:06:10.394099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:10.394071 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9661c8fb-1125-4c72-98f5-23f3c16bdebe-kube-api-access-tmrxt\" (UniqueName: \"kubernetes.io/projected/9661c8fb-1125-4c72-98f5-23f3c16bdebe-kube-api-access-tmrxt\") pod \"9661c8fb-1125-4c72-98f5-23f3c16bdebe\" (UID: \"9661c8fb-1125-4c72-98f5-23f3c16bdebe\") " Nov 02 15:06:10.396133 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:10.396106 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9661c8fb-1125-4c72-98f5-23f3c16bdebe-kube-api-access-tmrxt" pod "9661c8fb-1125-4c72-98f5-23f3c16bdebe" (UID: "9661c8fb-1125-4c72-98f5-23f3c16bdebe"). InnerVolumeSpecName "kube-api-access-tmrxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:10.494754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:10.494725 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrxt\" (UniqueName: \"kubernetes.io/projected/9661c8fb-1125-4c72-98f5-23f3c16bdebe-kube-api-access-tmrxt\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:06:11.010563 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:06:11.010525 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:06:11.106591 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:11.106553 5433 generic.go:356] "Generic (PLEG): container finished" podID="9661c8fb-1125-4c72-98f5-23f3c16bdebe" containerID="354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823" exitCode=0 Nov 02 15:06:11.106762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:11.106598 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-hngzt" Nov 02 15:06:11.106762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:11.106629 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hngzt" event={"ID":"9661c8fb-1125-4c72-98f5-23f3c16bdebe","Type":"ContainerDied","Data":"354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823"} Nov 02 15:06:11.106762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:11.106675 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-hngzt" event={"ID":"9661c8fb-1125-4c72-98f5-23f3c16bdebe","Type":"ContainerDied","Data":"4522aaf5d80910bd4de534f3315e6ebb46fd59c867e85d68424909a33ff77f46"} Nov 02 15:06:11.106762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:11.106686 5433 scope.go:122] "RemoveContainer" containerID="354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823" Nov 02 15:06:11.114772 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:11.114754 5433 scope.go:122] "RemoveContainer" containerID="354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823" Nov 02 15:06:11.115055 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:06:11.115026 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823\": container with ID starting with 354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823 not found: ID does not exist" containerID="354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823" Nov 02 15:06:11.115154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:11.115059 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823"} err="failed to get container status \"354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823\": rpc error: code = NotFound desc = could not find container \"354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823\": container with ID starting with 354be6219786d6ab89f886481e8c6d534b832e50488d1036b576461a2b812823 not found: ID does not exist" Nov 02 15:06:11.257452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:11.257412 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hngzt"] Nov 02 15:06:11.261798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:11.261712 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-hngzt"] Nov 02 15:06:12.453749 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:12.453717 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9661c8fb-1125-4c72-98f5-23f3c16bdebe" path="/var/lib/kubelet/pods/9661c8fb-1125-4c72-98f5-23f3c16bdebe/volumes" Nov 02 15:06:16.063372 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:16.063332 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:16.202145 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:16.202102 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:18.095195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:18.095166 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:18.508535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:18.508494 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:20.134764 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:20.134722 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f56cq"] Nov 02 15:06:20.135235 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:20.134951 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" podUID="117ef06d-ce68-4016-a18a-88db8b84cff0" containerName="limitador" containerID="cri-o://a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954" gracePeriod=30 Nov 02 15:06:20.677248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:20.677217 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:20.762257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:20.762168 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117ef06d-ce68-4016-a18a-88db8b84cff0-kube-api-access-klmzz\" (UniqueName: \"kubernetes.io/projected/117ef06d-ce68-4016-a18a-88db8b84cff0-kube-api-access-klmzz\") pod \"117ef06d-ce68-4016-a18a-88db8b84cff0\" (UID: \"117ef06d-ce68-4016-a18a-88db8b84cff0\") " Nov 02 15:06:20.762257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:20.762228 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117ef06d-ce68-4016-a18a-88db8b84cff0-config-file\" (UniqueName: \"kubernetes.io/configmap/117ef06d-ce68-4016-a18a-88db8b84cff0-config-file\") pod \"117ef06d-ce68-4016-a18a-88db8b84cff0\" (UID: \"117ef06d-ce68-4016-a18a-88db8b84cff0\") " Nov 02 15:06:20.762580 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:20.762559 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117ef06d-ce68-4016-a18a-88db8b84cff0-config-file" pod "117ef06d-ce68-4016-a18a-88db8b84cff0" (UID: "117ef06d-ce68-4016-a18a-88db8b84cff0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:06:20.764260 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:20.764237 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117ef06d-ce68-4016-a18a-88db8b84cff0-kube-api-access-klmzz" pod "117ef06d-ce68-4016-a18a-88db8b84cff0" (UID: "117ef06d-ce68-4016-a18a-88db8b84cff0"). InnerVolumeSpecName "kube-api-access-klmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:06:20.862745 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:20.862710 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klmzz\" (UniqueName: \"kubernetes.io/projected/117ef06d-ce68-4016-a18a-88db8b84cff0-kube-api-access-klmzz\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:06:20.862745 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:20.862739 5433 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/117ef06d-ce68-4016-a18a-88db8b84cff0-config-file\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:06:21.150903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:21.150865 5433 generic.go:356] "Generic (PLEG): container finished" podID="117ef06d-ce68-4016-a18a-88db8b84cff0" containerID="a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954" exitCode=0 Nov 02 15:06:21.151338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:21.150920 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" Nov 02 15:06:21.151338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:21.150947 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" event={"ID":"117ef06d-ce68-4016-a18a-88db8b84cff0","Type":"ContainerDied","Data":"a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954"} Nov 02 15:06:21.151338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:21.151024 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-f56cq" event={"ID":"117ef06d-ce68-4016-a18a-88db8b84cff0","Type":"ContainerDied","Data":"0c4089d7a627f4141f3d240a7b99081b02dcf4e5a31b3670963afc8baeda93cf"} Nov 02 15:06:21.151338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:21.151052 5433 scope.go:122] "RemoveContainer" containerID="a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954" Nov 02 15:06:21.159215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:21.159199 5433 scope.go:122] "RemoveContainer" containerID="a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954" Nov 02 15:06:21.159494 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:06:21.159472 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954\": container with ID starting with a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954 not found: ID does not exist" containerID="a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954" Nov 02 15:06:21.159549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:21.159504 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954"} err="failed to get container status \"a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954\": rpc error: code = NotFound desc = could not find container \"a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954\": container with ID starting with a790f433f999c8a9a0c30b9e6b71f036e88119115ad2e523f27fb722ce735954 not found: ID does not exist" Nov 02 15:06:21.302735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:21.302698 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f56cq"] Nov 02 15:06:21.308820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:21.308792 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-f56cq"] Nov 02 15:06:22.453721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:22.453689 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117ef06d-ce68-4016-a18a-88db8b84cff0" path="/var/lib/kubelet/pods/117ef06d-ce68-4016-a18a-88db8b84cff0/volumes" Nov 02 15:06:25.735566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:25.735517 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:25.879661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:25.879623 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:26.436338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.436302 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm"] Nov 02 15:06:26.436700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.436681 5433 cpu_manager.go:414] "containerMap: removing container" podUID="117ef06d-ce68-4016-a18a-88db8b84cff0" containerName="limitador" Nov 02 15:06:26.436700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.436700 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117ef06d-ce68-4016-a18a-88db8b84cff0" containerName="limitador" Nov 02 15:06:26.436870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.436728 5433 cpu_manager.go:414] "containerMap: removing container" podUID="9661c8fb-1125-4c72-98f5-23f3c16bdebe" containerName="authorino" Nov 02 15:06:26.436870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.436733 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9661c8fb-1125-4c72-98f5-23f3c16bdebe" containerName="authorino" Nov 02 15:06:26.436870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.436790 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9661c8fb-1125-4c72-98f5-23f3c16bdebe" containerName="authorino" Nov 02 15:06:26.436870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.436800 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="117ef06d-ce68-4016-a18a-88db8b84cff0" containerName="limitador" Nov 02 15:06:26.441681 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.441660 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:26.502413 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.502384 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ea0305c8-7c3e-46ca-a234-e3da552184bd-data\") pod \"postgres-7cdbf6dc44-bmbjm\" (UID: \"ea0305c8-7c3e-46ca-a234-e3da552184bd\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:26.502564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.502430 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67c5d\" (UniqueName: \"kubernetes.io/projected/ea0305c8-7c3e-46ca-a234-e3da552184bd-kube-api-access-67c5d\") pod \"postgres-7cdbf6dc44-bmbjm\" (UID: \"ea0305c8-7c3e-46ca-a234-e3da552184bd\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:26.504434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.504413 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 15:06:26.523042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.523012 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:26.536470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.536445 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:06:26.536470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.536454 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:06:26.536622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.536485 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ct54f\"" Nov 02 15:06:26.537280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.537261 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm"] Nov 02 15:06:26.603673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.603639 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ea0305c8-7c3e-46ca-a234-e3da552184bd-data\") pod \"postgres-7cdbf6dc44-bmbjm\" (UID: \"ea0305c8-7c3e-46ca-a234-e3da552184bd\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:26.603847 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.603683 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67c5d\" (UniqueName: \"kubernetes.io/projected/ea0305c8-7c3e-46ca-a234-e3da552184bd-kube-api-access-67c5d\") pod \"postgres-7cdbf6dc44-bmbjm\" (UID: \"ea0305c8-7c3e-46ca-a234-e3da552184bd\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:26.604135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.604114 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ea0305c8-7c3e-46ca-a234-e3da552184bd-data\") pod \"postgres-7cdbf6dc44-bmbjm\" (UID: \"ea0305c8-7c3e-46ca-a234-e3da552184bd\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:26.661134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.661108 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67c5d\" (UniqueName: \"kubernetes.io/projected/ea0305c8-7c3e-46ca-a234-e3da552184bd-kube-api-access-67c5d\") pod \"postgres-7cdbf6dc44-bmbjm\" (UID: \"ea0305c8-7c3e-46ca-a234-e3da552184bd\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:26.753214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:26.753133 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:27.113948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:27.113869 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm"] Nov 02 15:06:27.117231 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:06:27.117203 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0305c8_7c3e_46ca_a234_e3da552184bd.slice/crio-73527637ef3f1d9249618870f0f98fbb9f78ed1c3f08e568c144dd2d315e4ce5 WatchSource:0}: Error finding container 73527637ef3f1d9249618870f0f98fbb9f78ed1c3f08e568c144dd2d315e4ce5: Status 404 returned error can't find the container with id 73527637ef3f1d9249618870f0f98fbb9f78ed1c3f08e568c144dd2d315e4ce5 Nov 02 15:06:27.174331 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:27.174294 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" event={"ID":"ea0305c8-7c3e-46ca-a234-e3da552184bd","Type":"ContainerStarted","Data":"73527637ef3f1d9249618870f0f98fbb9f78ed1c3f08e568c144dd2d315e4ce5"} Nov 02 15:06:34.210762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:34.210726 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" event={"ID":"ea0305c8-7c3e-46ca-a234-e3da552184bd","Type":"ContainerStarted","Data":"86e9359fb67528beca7e172fa0e8ee95ca6c529ec29600717d9d22696a8b8ddf"} Nov 02 15:06:34.211274 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:34.210806 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:34.318154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:34.318097 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" podStartSLOduration=1.769369084 podStartE2EDuration="8.318084798s" totalImagesPullingTime="6.548715714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:34.31804652 +0000 UTC m=+497.991249716" watchObservedRunningTime="2026-11-02 15:06:34.318084798 +0000 UTC m=+497.991287993" Nov 02 15:06:40.241843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:40.241812 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bmbjm" Nov 02 15:06:41.009796 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:06:41.009760 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:06:47.714121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.714088 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:47.788749 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.788717 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-kdt5p"] Nov 02 15:06:47.794076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.794051 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:47.842609 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.842578 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 15:06:47.868640 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.868611 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:47.877926 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.877892 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps7hg\" (UniqueName: \"kubernetes.io/projected/1f092c35-c986-4860-8cc6-483da5792667-kube-api-access-ps7hg\") pod \"maas-controller-579cf7c6d-kdt5p\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:47.878090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.877947 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-webhook-cert\") pod \"maas-controller-579cf7c6d-kdt5p\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:47.878090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.878067 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-metrics-certs\") pod \"maas-controller-579cf7c6d-kdt5p\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:47.888607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.888584 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8b769\"" Nov 02 15:06:47.888765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.888645 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 15:06:47.888765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.888700 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 15:06:47.888765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.888726 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 15:06:47.888765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.888734 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 15:06:47.889536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.889518 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-kdt5p"] Nov 02 15:06:47.979442 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.979369 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps7hg\" (UniqueName: \"kubernetes.io/projected/1f092c35-c986-4860-8cc6-483da5792667-kube-api-access-ps7hg\") pod \"maas-controller-579cf7c6d-kdt5p\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:47.979442 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.979419 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-webhook-cert\") pod \"maas-controller-579cf7c6d-kdt5p\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:47.979628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.979454 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-metrics-certs\") pod \"maas-controller-579cf7c6d-kdt5p\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:47.982008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.981970 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-webhook-cert\") pod \"maas-controller-579cf7c6d-kdt5p\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:47.982130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:47.982040 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-metrics-certs\") pod \"maas-controller-579cf7c6d-kdt5p\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:48.016016 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.015947 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:48.036809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.036783 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps7hg\" (UniqueName: \"kubernetes.io/projected/1f092c35-c986-4860-8cc6-483da5792667-kube-api-access-ps7hg\") pod \"maas-controller-579cf7c6d-kdt5p\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:48.106153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.106116 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:48.168634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.168605 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:48.223992 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.223947 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-754d6b6455-5rk2q"] Nov 02 15:06:48.226690 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.226666 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.282922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.282887 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-webhook-cert\") pod \"maas-controller-754d6b6455-5rk2q\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.283140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.282939 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-metrics-certs\") pod \"maas-controller-754d6b6455-5rk2q\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.283140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.283053 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9rh\" (UniqueName: \"kubernetes.io/projected/71a350d7-03c8-45e7-ac2a-de0be80d6855-kube-api-access-wd9rh\") pod \"maas-controller-754d6b6455-5rk2q\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.285305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.285280 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-754d6b6455-5rk2q"] Nov 02 15:06:48.384192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.384150 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-metrics-certs\") pod \"maas-controller-754d6b6455-5rk2q\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.384373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.384205 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9rh\" (UniqueName: \"kubernetes.io/projected/71a350d7-03c8-45e7-ac2a-de0be80d6855-kube-api-access-wd9rh\") pod \"maas-controller-754d6b6455-5rk2q\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.384373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.384289 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-webhook-cert\") pod \"maas-controller-754d6b6455-5rk2q\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.386736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.386708 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-webhook-cert\") pod \"maas-controller-754d6b6455-5rk2q\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.386736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.386732 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-metrics-certs\") pod \"maas-controller-754d6b6455-5rk2q\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.456930 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.456895 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9rh\" (UniqueName: \"kubernetes.io/projected/71a350d7-03c8-45e7-ac2a-de0be80d6855-kube-api-access-wd9rh\") pod \"maas-controller-754d6b6455-5rk2q\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.491560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.489866 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-kdt5p"] Nov 02 15:06:48.492318 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:06:48.492289 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f092c35_c986_4860_8cc6_483da5792667.slice/crio-ce7446075ddbfa044f8c12353bc0996b234720eb9ac5f05855dee62d6cb4564b WatchSource:0}: Error finding container ce7446075ddbfa044f8c12353bc0996b234720eb9ac5f05855dee62d6cb4564b: Status 404 returned error can't find the container with id ce7446075ddbfa044f8c12353bc0996b234720eb9ac5f05855dee62d6cb4564b Nov 02 15:06:48.539485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.539446 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:48.917068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:48.917035 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-754d6b6455-5rk2q"] Nov 02 15:06:48.920583 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:06:48.920533 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a350d7_03c8_45e7_ac2a_de0be80d6855.slice/crio-6a90091a9838cb602bbb21c8d4e7eb15488962a5bc017de7fec2d6776d0f2b48 WatchSource:0}: Error finding container 6a90091a9838cb602bbb21c8d4e7eb15488962a5bc017de7fec2d6776d0f2b48: Status 404 returned error can't find the container with id 6a90091a9838cb602bbb21c8d4e7eb15488962a5bc017de7fec2d6776d0f2b48 Nov 02 15:06:49.274095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:49.274004 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" event={"ID":"1f092c35-c986-4860-8cc6-483da5792667","Type":"ContainerStarted","Data":"ce7446075ddbfa044f8c12353bc0996b234720eb9ac5f05855dee62d6cb4564b"} Nov 02 15:06:49.275251 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:49.275210 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-5rk2q" event={"ID":"71a350d7-03c8-45e7-ac2a-de0be80d6855","Type":"ContainerStarted","Data":"6a90091a9838cb602bbb21c8d4e7eb15488962a5bc017de7fec2d6776d0f2b48"} Nov 02 15:06:52.690343 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:52.690311 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:52.853584 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:52.853535 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:53.295931 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:53.295886 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" event={"ID":"1f092c35-c986-4860-8cc6-483da5792667","Type":"ContainerStarted","Data":"7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642"} Nov 02 15:06:53.296146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:53.295960 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:06:53.297440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:53.297407 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-5rk2q" event={"ID":"71a350d7-03c8-45e7-ac2a-de0be80d6855","Type":"ContainerStarted","Data":"85e43fb1fb182e84f4b8ee5a57ee69439456b471a5fc2ffe0ab130f32407ef1d"} Nov 02 15:06:53.297556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:53.297535 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:06:53.420883 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:53.420828 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" podStartSLOduration=2.662526905 podStartE2EDuration="6.420808901s" totalImagesPullingTime="3.758281996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:53.420280153 +0000 UTC m=+517.093483348" watchObservedRunningTime="2026-11-02 15:06:53.420808901 +0000 UTC m=+517.094012098" Nov 02 15:06:53.531243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:53.531177 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-754d6b6455-5rk2q" podStartSLOduration=2.137440017 podStartE2EDuration="5.531163773s" totalImagesPullingTime="3.393723756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:53.531043936 +0000 UTC m=+517.204247130" watchObservedRunningTime="2026-11-02 15:06:53.531163773 +0000 UTC m=+517.204366968" Nov 02 15:06:57.260281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.260242 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c"] Nov 02 15:06:57.263036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.263015 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.364576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.364529 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c"] Nov 02 15:06:57.364751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.364649 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f44d6239-c49c-4d30-839f-77bcf9eea65b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.364751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.364699 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.364751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.364742 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44d6239-c49c-4d30-839f-77bcf9eea65b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.364906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.364830 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.364906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.364862 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nshgh\" (UniqueName: \"kubernetes.io/projected/f44d6239-c49c-4d30-839f-77bcf9eea65b-kube-api-access-nshgh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.364906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.364882 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.364906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.364904 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.466367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.466324 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f44d6239-c49c-4d30-839f-77bcf9eea65b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.466556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.466420 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.466556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.466469 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44d6239-c49c-4d30-839f-77bcf9eea65b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.466556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.466510 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.466790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.466555 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nshgh\" (UniqueName: \"kubernetes.io/projected/f44d6239-c49c-4d30-839f-77bcf9eea65b-kube-api-access-nshgh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.466790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.466584 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.466790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.466729 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.467759 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.467728 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.469788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.469733 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f44d6239-c49c-4d30-839f-77bcf9eea65b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.469788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.469758 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.469950 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.469871 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.470240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.470217 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f44d6239-c49c-4d30-839f-77bcf9eea65b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.527037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.526950 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f44d6239-c49c-4d30-839f-77bcf9eea65b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.527468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.527443 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nshgh\" (UniqueName: \"kubernetes.io/projected/f44d6239-c49c-4d30-839f-77bcf9eea65b-kube-api-access-nshgh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5lx6c\" (UID: \"f44d6239-c49c-4d30-839f-77bcf9eea65b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.574176 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.574143 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:57.720313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.720270 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:57.868064 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:57.867961 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:06:58.005010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:58.004965 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c"] Nov 02 15:06:58.008123 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:06:58.008096 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44d6239_c49c_4d30_839f_77bcf9eea65b.slice/crio-fe35ec7ad260e468eed665801c72d7a3927c05e4f224be75bad3795b31186f13 WatchSource:0}: Error finding container fe35ec7ad260e468eed665801c72d7a3927c05e4f224be75bad3795b31186f13: Status 404 returned error can't find the container with id fe35ec7ad260e468eed665801c72d7a3927c05e4f224be75bad3795b31186f13 Nov 02 15:06:58.010158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:58.010132 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:06:58.010229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:58.010186 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:06:58.318476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:58.318438 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" event={"ID":"f44d6239-c49c-4d30-839f-77bcf9eea65b","Type":"ContainerStarted","Data":"87087198c3604fd905fc2c4ede12ddedf6e4fdd047c21e175220eede9145d8ea"} Nov 02 15:06:58.319005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:58.318494 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" event={"ID":"f44d6239-c49c-4d30-839f-77bcf9eea65b","Type":"ContainerStarted","Data":"fe35ec7ad260e468eed665801c72d7a3927c05e4f224be75bad3795b31186f13"} Nov 02 15:06:58.319005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:58.318556 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:06:58.435431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:58.435374 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" podStartSLOduration=1.435360805 podStartE2EDuration="1.435360805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:58.434166521 +0000 UTC m=+522.107369719" watchObservedRunningTime="2026-11-02 15:06:58.435360805 +0000 UTC m=+522.108564000" Nov 02 15:06:59.324754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:59.324724 5433 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5lx6c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:06:59.325167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:06:59.324781 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" podUID="f44d6239-c49c-4d30-839f-77bcf9eea65b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:07:01.617059 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:01.617029 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:01.767899 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:01.767866 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:01.936494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:01.936458 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:02.147840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.147798 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:02.321026 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.320913 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:02.324435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.324414 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5lx6c" Nov 02 15:07:02.480481 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.480418 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:02.490654 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.490624 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww"] Nov 02 15:07:02.490886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.490836 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" podUID="20cef6be-a6dc-443d-91f2-ab4a863b8779" containerName="discovery" containerID="cri-o://d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e" gracePeriod=30 Nov 02 15:07:02.652278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.652237 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:02.732511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.732481 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:07:02.812675 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.812643 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cef6be-a6dc-443d-91f2-ab4a863b8779-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef6be-a6dc-443d-91f2-ab4a863b8779-local-certs\") pod \"20cef6be-a6dc-443d-91f2-ab4a863b8779\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " Nov 02 15:07:02.812856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.812708 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-kube-api-access-f5jtj\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-kube-api-access-f5jtj\") pod \"20cef6be-a6dc-443d-91f2-ab4a863b8779\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " Nov 02 15:07:02.812856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.812731 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-kubeconfig\") pod \"20cef6be-a6dc-443d-91f2-ab4a863b8779\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " Nov 02 15:07:02.812856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.812768 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-dns-cert\") pod \"20cef6be-a6dc-443d-91f2-ab4a863b8779\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " Nov 02 15:07:02.812856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.812804 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-cacerts\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-cacerts\") pod \"20cef6be-a6dc-443d-91f2-ab4a863b8779\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " Nov 02 15:07:02.812856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.812821 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-token\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-token\") pod \"20cef6be-a6dc-443d-91f2-ab4a863b8779\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " Nov 02 15:07:02.812856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.812841 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-ca-configmap\") pod \"20cef6be-a6dc-443d-91f2-ab4a863b8779\" (UID: \"20cef6be-a6dc-443d-91f2-ab4a863b8779\") " Nov 02 15:07:02.813429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.813395 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-ca-configmap" pod "20cef6be-a6dc-443d-91f2-ab4a863b8779" (UID: "20cef6be-a6dc-443d-91f2-ab4a863b8779"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:07:02.815087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.815060 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-token" pod "20cef6be-a6dc-443d-91f2-ab4a863b8779" (UID: "20cef6be-a6dc-443d-91f2-ab4a863b8779"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:02.815200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.815161 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cef6be-a6dc-443d-91f2-ab4a863b8779-local-certs" pod "20cef6be-a6dc-443d-91f2-ab4a863b8779" (UID: "20cef6be-a6dc-443d-91f2-ab4a863b8779"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:07:02.815444 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.815417 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-kube-api-access-f5jtj" pod "20cef6be-a6dc-443d-91f2-ab4a863b8779" (UID: "20cef6be-a6dc-443d-91f2-ab4a863b8779"). InnerVolumeSpecName "kube-api-access-f5jtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:02.815534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.815459 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-kubeconfig" pod "20cef6be-a6dc-443d-91f2-ab4a863b8779" (UID: "20cef6be-a6dc-443d-91f2-ab4a863b8779"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:02.815534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.815481 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-dns-cert" pod "20cef6be-a6dc-443d-91f2-ab4a863b8779" (UID: "20cef6be-a6dc-443d-91f2-ab4a863b8779"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:02.815534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.815498 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-cacerts" pod "20cef6be-a6dc-443d-91f2-ab4a863b8779" (UID: "20cef6be-a6dc-443d-91f2-ab4a863b8779"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:02.876948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.876911 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-bs4qx"] Nov 02 15:07:02.877303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.877288 5433 cpu_manager.go:414] "containerMap: removing container" podUID="20cef6be-a6dc-443d-91f2-ab4a863b8779" containerName="discovery" Nov 02 15:07:02.877355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.877305 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20cef6be-a6dc-443d-91f2-ab4a863b8779" containerName="discovery" Nov 02 15:07:02.877389 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.877374 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="20cef6be-a6dc-443d-91f2-ab4a863b8779" containerName="discovery" Nov 02 15:07:02.879636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.879617 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:02.913885 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.913861 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5jtj\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-kube-api-access-f5jtj\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:02.913885 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.913884 5433 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-kubeconfig\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:02.914080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.913895 5433 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-dns-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:02.914080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.913905 5433 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20cef6be-a6dc-443d-91f2-ab4a863b8779-cacerts\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:02.914080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.913915 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:02.914080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.913925 5433 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20cef6be-a6dc-443d-91f2-ab4a863b8779-istio-csr-ca-configmap\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:02.914080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.913934 5433 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cef6be-a6dc-443d-91f2-ab4a863b8779-local-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:02.928729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.928706 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 15:07:02.928729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.928706 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 15:07:02.928885 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.928707 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lhccp\"" Nov 02 15:07:02.935756 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:02.935726 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bs4qx"] Nov 02 15:07:03.014629 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.014589 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5wqg\" (UniqueName: \"kubernetes.io/projected/ef257c3c-af16-4b85-bfa1-3208f6b0f69e-kube-api-access-w5wqg\") pod \"odh-model-controller-769875db56-bs4qx\" (UID: \"ef257c3c-af16-4b85-bfa1-3208f6b0f69e\") " pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.014815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.014643 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef257c3c-af16-4b85-bfa1-3208f6b0f69e-metrics-certs\") pod \"odh-model-controller-769875db56-bs4qx\" (UID: \"ef257c3c-af16-4b85-bfa1-3208f6b0f69e\") " pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.014815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.014721 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef257c3c-af16-4b85-bfa1-3208f6b0f69e-cert\") pod \"odh-model-controller-769875db56-bs4qx\" (UID: \"ef257c3c-af16-4b85-bfa1-3208f6b0f69e\") " pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.115519 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.115477 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef257c3c-af16-4b85-bfa1-3208f6b0f69e-metrics-certs\") pod \"odh-model-controller-769875db56-bs4qx\" (UID: \"ef257c3c-af16-4b85-bfa1-3208f6b0f69e\") " pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.115519 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.115520 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef257c3c-af16-4b85-bfa1-3208f6b0f69e-cert\") pod \"odh-model-controller-769875db56-bs4qx\" (UID: \"ef257c3c-af16-4b85-bfa1-3208f6b0f69e\") " pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.115724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.115699 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5wqg\" (UniqueName: \"kubernetes.io/projected/ef257c3c-af16-4b85-bfa1-3208f6b0f69e-kube-api-access-w5wqg\") pod \"odh-model-controller-769875db56-bs4qx\" (UID: \"ef257c3c-af16-4b85-bfa1-3208f6b0f69e\") " pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.117986 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.117942 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef257c3c-af16-4b85-bfa1-3208f6b0f69e-metrics-certs\") pod \"odh-model-controller-769875db56-bs4qx\" (UID: \"ef257c3c-af16-4b85-bfa1-3208f6b0f69e\") " pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.118105 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.117956 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef257c3c-af16-4b85-bfa1-3208f6b0f69e-cert\") pod \"odh-model-controller-769875db56-bs4qx\" (UID: \"ef257c3c-af16-4b85-bfa1-3208f6b0f69e\") " pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.172892 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.172821 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5wqg\" (UniqueName: \"kubernetes.io/projected/ef257c3c-af16-4b85-bfa1-3208f6b0f69e-kube-api-access-w5wqg\") pod \"odh-model-controller-769875db56-bs4qx\" (UID: \"ef257c3c-af16-4b85-bfa1-3208f6b0f69e\") " pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.191697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.191669 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:03.339562 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.339526 5433 generic.go:356] "Generic (PLEG): container finished" podID="20cef6be-a6dc-443d-91f2-ab4a863b8779" containerID="d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e" exitCode=0 Nov 02 15:07:03.339737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.339582 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" event={"ID":"20cef6be-a6dc-443d-91f2-ab4a863b8779","Type":"ContainerDied","Data":"d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e"} Nov 02 15:07:03.339737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.339622 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" event={"ID":"20cef6be-a6dc-443d-91f2-ab4a863b8779","Type":"ContainerDied","Data":"18a6c6bd9cd83b6290d1cecd3812d0888d9d5483e6cc8423e644b86fc71c622f"} Nov 02 15:07:03.339737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.339642 5433 scope.go:122] "RemoveContainer" containerID="d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e" Nov 02 15:07:03.339737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.339582 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww" Nov 02 15:07:03.348476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.348452 5433 scope.go:122] "RemoveContainer" containerID="d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e" Nov 02 15:07:03.348772 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:07:03.348750 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e\": container with ID starting with d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e not found: ID does not exist" containerID="d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e" Nov 02 15:07:03.348836 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.348786 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e"} err="failed to get container status \"d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e\": rpc error: code = NotFound desc = could not find container \"d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e\": container with ID starting with d61a0f9e552c4b51ce72aeeae441f275756fd0cd30f3bb0bc3c656b98c16290e not found: ID does not exist" Nov 02 15:07:03.493005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.492918 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww"] Nov 02 15:07:03.496755 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.496726 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jzdww"] Nov 02 15:07:03.599713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.599677 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-bs4qx"] Nov 02 15:07:03.603215 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:07:03.603175 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef257c3c_af16_4b85_bfa1_3208f6b0f69e.slice/crio-ff897785fe6be28cdecc18d2b9daeb8de07f7485e55b03aeac6f281121e854ec WatchSource:0}: Error finding container ff897785fe6be28cdecc18d2b9daeb8de07f7485e55b03aeac6f281121e854ec: Status 404 returned error can't find the container with id ff897785fe6be28cdecc18d2b9daeb8de07f7485e55b03aeac6f281121e854ec Nov 02 15:07:03.865370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:03.865294 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:04.303759 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.303723 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:07:04.303909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.303799 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:07:04.345094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.345059 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bs4qx" event={"ID":"ef257c3c-af16-4b85-bfa1-3208f6b0f69e","Type":"ContainerStarted","Data":"ff897785fe6be28cdecc18d2b9daeb8de07f7485e55b03aeac6f281121e854ec"} Nov 02 15:07:04.456218 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.456190 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20cef6be-a6dc-443d-91f2-ab4a863b8779" path="/var/lib/kubelet/pods/20cef6be-a6dc-443d-91f2-ab4a863b8779/volumes" Nov 02 15:07:04.463919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.463891 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-kdt5p"] Nov 02 15:07:04.464143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.464104 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" podUID="1f092c35-c986-4860-8cc6-483da5792667" containerName="manager" containerID="cri-o://7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642" gracePeriod=30 Nov 02 15:07:04.754308 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.754285 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:07:04.828835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.828800 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-webhook-cert\") pod \"1f092c35-c986-4860-8cc6-483da5792667\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " Nov 02 15:07:04.828969 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.828874 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-metrics-certs\") pod \"1f092c35-c986-4860-8cc6-483da5792667\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " Nov 02 15:07:04.828969 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.828964 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f092c35-c986-4860-8cc6-483da5792667-kube-api-access-ps7hg\" (UniqueName: \"kubernetes.io/projected/1f092c35-c986-4860-8cc6-483da5792667-kube-api-access-ps7hg\") pod \"1f092c35-c986-4860-8cc6-483da5792667\" (UID: \"1f092c35-c986-4860-8cc6-483da5792667\") " Nov 02 15:07:04.831790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.831749 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-webhook-cert" pod "1f092c35-c986-4860-8cc6-483da5792667" (UID: "1f092c35-c986-4860-8cc6-483da5792667"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:04.831899 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.831866 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f092c35-c986-4860-8cc6-483da5792667-kube-api-access-ps7hg" pod "1f092c35-c986-4860-8cc6-483da5792667" (UID: "1f092c35-c986-4860-8cc6-483da5792667"). InnerVolumeSpecName "kube-api-access-ps7hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:04.833158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.833123 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-metrics-certs" pod "1f092c35-c986-4860-8cc6-483da5792667" (UID: "1f092c35-c986-4860-8cc6-483da5792667"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:04.930224 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.930186 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps7hg\" (UniqueName: \"kubernetes.io/projected/1f092c35-c986-4860-8cc6-483da5792667-kube-api-access-ps7hg\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:04.930224 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.930220 5433 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-webhook-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:04.930764 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:04.930232 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f092c35-c986-4860-8cc6-483da5792667-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:05.352509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:05.352410 5433 generic.go:356] "Generic (PLEG): container finished" podID="1f092c35-c986-4860-8cc6-483da5792667" containerID="7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642" exitCode=0 Nov 02 15:07:05.352509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:05.352489 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" Nov 02 15:07:05.352737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:05.352523 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" event={"ID":"1f092c35-c986-4860-8cc6-483da5792667","Type":"ContainerDied","Data":"7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642"} Nov 02 15:07:05.352737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:05.352600 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-kdt5p" event={"ID":"1f092c35-c986-4860-8cc6-483da5792667","Type":"ContainerDied","Data":"ce7446075ddbfa044f8c12353bc0996b234720eb9ac5f05855dee62d6cb4564b"} Nov 02 15:07:05.352737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:05.352630 5433 scope.go:122] "RemoveContainer" containerID="7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642" Nov 02 15:07:05.364876 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:05.364850 5433 scope.go:122] "RemoveContainer" containerID="7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642" Nov 02 15:07:05.365286 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:07:05.365214 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642\": container with ID starting with 7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642 not found: ID does not exist" containerID="7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642" Nov 02 15:07:05.365286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:05.365256 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642"} err="failed to get container status \"7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642\": rpc error: code = NotFound desc = could not find container \"7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642\": container with ID starting with 7988e17e38f9ade2cf90c26adc121d21105bb638f81d984c852da6607edec642 not found: ID does not exist" Nov 02 15:07:05.505477 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:05.505438 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-kdt5p"] Nov 02 15:07:05.551200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:05.551161 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-kdt5p"] Nov 02 15:07:06.454961 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:06.454926 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f092c35-c986-4860-8cc6-483da5792667" path="/var/lib/kubelet/pods/1f092c35-c986-4860-8cc6-483da5792667/volumes" Nov 02 15:07:07.364285 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:07.364248 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-bs4qx" event={"ID":"ef257c3c-af16-4b85-bfa1-3208f6b0f69e","Type":"ContainerStarted","Data":"b31cc66a6159cc509af49d78ae2ad4b85a5a9b78e15e1c74b7837e59cd9ef894"} Nov 02 15:07:07.364460 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:07.364374 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:07.474781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:07.474724 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-bs4qx" podStartSLOduration=2.2613856119999998 podStartE2EDuration="5.474706979s" totalImagesPullingTime="3.213321367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:07.473873336 +0000 UTC m=+531.147076534" watchObservedRunningTime="2026-11-02 15:07:07.474706979 +0000 UTC m=+531.147910174" Nov 02 15:07:08.701453 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:08.701414 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:08.849406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:08.849370 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:08.995928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:08.995847 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:09.146292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:09.146253 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:09.288767 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:09.288686 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:09.440499 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:09.440463 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:09.622509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:09.622402 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:09.768170 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:09.768132 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:09.923482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:09.923447 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:11.010718 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:07:11.010616 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:07:18.371257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:18.371213 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-bs4qx" Nov 02 15:07:18.847844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:18.847814 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:26.528427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:26.528341 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:26.672941 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:26.672905 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:28.749336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:28.749300 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:28.922944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:28.922909 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:29.071258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.071166 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:29.220896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.220859 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:29.799037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.798996 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj"] Nov 02 15:07:29.799543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.799484 5433 cpu_manager.go:414] "containerMap: removing container" podUID="1f092c35-c986-4860-8cc6-483da5792667" containerName="manager" Nov 02 15:07:29.799543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.799505 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f092c35-c986-4860-8cc6-483da5792667" containerName="manager" Nov 02 15:07:29.799717 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.799621 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f092c35-c986-4860-8cc6-483da5792667" containerName="manager" Nov 02 15:07:29.802631 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.802610 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.839908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.839879 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfxq4\" (UniqueName: \"kubernetes.io/projected/b077df3c-3329-42c3-853e-d789ecd2cbe5-kube-api-access-tfxq4\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.839908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.839910 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-metrics-certs\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.840136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.839930 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b077df3c-3329-42c3-853e-d789ecd2cbe5-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.840136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.840043 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-maas-api-tls\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.898928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.898884 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 15:07:29.898928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.898886 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:07:29.898928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.898886 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wj8b6\"" Nov 02 15:07:29.899874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.899852 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj"] Nov 02 15:07:29.941236 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.941200 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-maas-api-tls\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.941421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.941272 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxq4\" (UniqueName: \"kubernetes.io/projected/b077df3c-3329-42c3-853e-d789ecd2cbe5-kube-api-access-tfxq4\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.941421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.941291 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-metrics-certs\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.941421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.941312 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b077df3c-3329-42c3-853e-d789ecd2cbe5-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.942007 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.941965 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b077df3c-3329-42c3-853e-d789ecd2cbe5-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.943806 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.943779 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-maas-api-tls\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.943901 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.943809 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-metrics-certs\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:29.998635 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:29.998601 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxq4\" (UniqueName: \"kubernetes.io/projected/b077df3c-3329-42c3-853e-d789ecd2cbe5-kube-api-access-tfxq4\") pod \"maas-api-6d669c7fcd-fprfj\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:30.114947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:30.114853 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:30.481924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:30.481892 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj"] Nov 02 15:07:30.485154 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:07:30.485131 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb077df3c_3329_42c3_853e_d789ecd2cbe5.slice/crio-bc87599f8c3cced29193b8182dca98c83f29eb3e49e778bfb6d0460d09a84959 WatchSource:0}: Error finding container bc87599f8c3cced29193b8182dca98c83f29eb3e49e778bfb6d0460d09a84959: Status 404 returned error can't find the container with id bc87599f8c3cced29193b8182dca98c83f29eb3e49e778bfb6d0460d09a84959 Nov 02 15:07:30.796995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:30.796895 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:30.941840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:30.941804 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:31.458550 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:31.458518 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" event={"ID":"b077df3c-3329-42c3-853e-d789ecd2cbe5","Type":"ContainerStarted","Data":"bc87599f8c3cced29193b8182dca98c83f29eb3e49e778bfb6d0460d09a84959"} Nov 02 15:07:34.472233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:34.472194 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" event={"ID":"b077df3c-3329-42c3-853e-d789ecd2cbe5","Type":"ContainerStarted","Data":"01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac"} Nov 02 15:07:34.472616 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:34.472333 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:34.583269 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:34.583206 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" podStartSLOduration=2.199261427 podStartE2EDuration="5.583187367s" totalImagesPullingTime="3.38392594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:34.582616779 +0000 UTC m=+558.255819975" watchObservedRunningTime="2026-11-02 15:07:34.583187367 +0000 UTC m=+558.256390562" Nov 02 15:07:40.481651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:40.481620 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:41.002108 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.002076 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:41.010176 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:07:41.010146 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:07:41.292306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.292215 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h65g2"] Nov 02 15:07:41.297426 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.297401 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" Nov 02 15:07:41.301015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.300880 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:07:41.342210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.342174 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvqh\" (UniqueName: \"kubernetes.io/projected/2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0-kube-api-access-kbvqh\") pod \"authorino-6f6944bcdd-h65g2\" (UID: \"2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0\") " pod="kuadrant-system/authorino-6f6944bcdd-h65g2" Nov 02 15:07:41.349774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.349744 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h65g2"] Nov 02 15:07:41.392113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.392079 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 15:07:41.392113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.392086 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 15:07:41.392318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.392178 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5vb2v\"" Nov 02 15:07:41.443414 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.443377 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvqh\" (UniqueName: \"kubernetes.io/projected/2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0-kube-api-access-kbvqh\") pod \"authorino-6f6944bcdd-h65g2\" (UID: \"2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0\") " pod="kuadrant-system/authorino-6f6944bcdd-h65g2" Nov 02 15:07:41.473234 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.473167 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-nj427"] Nov 02 15:07:41.476124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.476102 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.501226 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.501197 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvqh\" (UniqueName: \"kubernetes.io/projected/2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0-kube-api-access-kbvqh\") pod \"authorino-6f6944bcdd-h65g2\" (UID: \"2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0\") " pod="kuadrant-system/authorino-6f6944bcdd-h65g2" Nov 02 15:07:41.534392 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.534355 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-nj427"] Nov 02 15:07:41.544611 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.544542 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010ec615-c5e2-4791-94a8-3951ce25ec1a-openshift-service-ca\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.544747 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.544725 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-maas-api-tls\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.544851 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.544826 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-metrics-certs\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.544925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.544864 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm5jp\" (UniqueName: \"kubernetes.io/projected/010ec615-c5e2-4791-94a8-3951ce25ec1a-kube-api-access-wm5jp\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.608939 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.608902 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" Nov 02 15:07:41.647128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.646345 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010ec615-c5e2-4791-94a8-3951ce25ec1a-openshift-service-ca\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.647330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.647247 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010ec615-c5e2-4791-94a8-3951ce25ec1a-openshift-service-ca\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.647420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.647383 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-maas-api-tls\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.647468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.647449 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-metrics-certs\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.647524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.647475 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm5jp\" (UniqueName: \"kubernetes.io/projected/010ec615-c5e2-4791-94a8-3951ce25ec1a-kube-api-access-wm5jp\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.650093 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.650069 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-maas-api-tls\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.650194 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.650136 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-metrics-certs\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.663549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.663378 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h65g2"] Nov 02 15:07:41.690490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.690240 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5588dd55b9-j4jbw"] Nov 02 15:07:41.693798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.693647 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" Nov 02 15:07:41.706790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.706759 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm5jp\" (UniqueName: \"kubernetes.io/projected/010ec615-c5e2-4791-94a8-3951ce25ec1a-kube-api-access-wm5jp\") pod \"maas-api-74cdd4d67-nj427\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.747379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.747344 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5588dd55b9-j4jbw"] Nov 02 15:07:41.748873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.748833 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xkq2\" (UniqueName: \"kubernetes.io/projected/60b43ccc-6f07-4876-9f42-1c55eae77c3c-kube-api-access-5xkq2\") pod \"authorino-5588dd55b9-j4jbw\" (UID: \"60b43ccc-6f07-4876-9f42-1c55eae77c3c\") " pod="kuadrant-system/authorino-5588dd55b9-j4jbw" Nov 02 15:07:41.787843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.787813 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:41.849576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.849547 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkq2\" (UniqueName: \"kubernetes.io/projected/60b43ccc-6f07-4876-9f42-1c55eae77c3c-kube-api-access-5xkq2\") pod \"authorino-5588dd55b9-j4jbw\" (UID: \"60b43ccc-6f07-4876-9f42-1c55eae77c3c\") " pod="kuadrant-system/authorino-5588dd55b9-j4jbw" Nov 02 15:07:41.906958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.906924 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkq2\" (UniqueName: \"kubernetes.io/projected/60b43ccc-6f07-4876-9f42-1c55eae77c3c-kube-api-access-5xkq2\") pod \"authorino-5588dd55b9-j4jbw\" (UID: \"60b43ccc-6f07-4876-9f42-1c55eae77c3c\") " pod="kuadrant-system/authorino-5588dd55b9-j4jbw" Nov 02 15:07:41.974671 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:41.974634 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h65g2"] Nov 02 15:07:41.977873 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:07:41.977845 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bb47aa9_c39f_4c57_ad18_42a3acc6ffb0.slice/crio-656d5682a9c8748b032833ee181f6e8ed9742fc375fd65a5066849f4258aaf91 WatchSource:0}: Error finding container 656d5682a9c8748b032833ee181f6e8ed9742fc375fd65a5066849f4258aaf91: Status 404 returned error can't find the container with id 656d5682a9c8748b032833ee181f6e8ed9742fc375fd65a5066849f4258aaf91 Nov 02 15:07:42.011046 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.011021 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" Nov 02 15:07:42.024146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.024117 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5588dd55b9-j4jbw"] Nov 02 15:07:42.056051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.055304 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b89d89487-gvj4d"] Nov 02 15:07:42.059110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.059086 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:07:42.108240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.108214 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 15:07:42.113371 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.113334 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b89d89487-gvj4d"] Nov 02 15:07:42.149205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.149174 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-nj427"] Nov 02 15:07:42.152097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.152069 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvv4\" (UniqueName: \"kubernetes.io/projected/d43b6840-ec0e-4929-bd0e-ba89875d57cb-kube-api-access-jsvv4\") pod \"authorino-7b89d89487-gvj4d\" (UID: \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\") " pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:07:42.152236 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.152134 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d43b6840-ec0e-4929-bd0e-ba89875d57cb-tls-cert\") pod \"authorino-7b89d89487-gvj4d\" (UID: \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\") " pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:07:42.152236 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:07:42.152213 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010ec615_c5e2_4791_94a8_3951ce25ec1a.slice/crio-c5ea29abade11f85003fafc87b6354d9b5c491aba9601421c302ff16ca1e0b55 WatchSource:0}: Error finding container c5ea29abade11f85003fafc87b6354d9b5c491aba9601421c302ff16ca1e0b55: Status 404 returned error can't find the container with id c5ea29abade11f85003fafc87b6354d9b5c491aba9601421c302ff16ca1e0b55 Nov 02 15:07:42.253479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.253445 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvv4\" (UniqueName: \"kubernetes.io/projected/d43b6840-ec0e-4929-bd0e-ba89875d57cb-kube-api-access-jsvv4\") pod \"authorino-7b89d89487-gvj4d\" (UID: \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\") " pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:07:42.253661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.253503 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d43b6840-ec0e-4929-bd0e-ba89875d57cb-tls-cert\") pod \"authorino-7b89d89487-gvj4d\" (UID: \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\") " pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:07:42.255918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.255889 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d43b6840-ec0e-4929-bd0e-ba89875d57cb-tls-cert\") pod \"authorino-7b89d89487-gvj4d\" (UID: \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\") " pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:07:42.310626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.310553 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvv4\" (UniqueName: \"kubernetes.io/projected/d43b6840-ec0e-4929-bd0e-ba89875d57cb-kube-api-access-jsvv4\") pod \"authorino-7b89d89487-gvj4d\" (UID: \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\") " pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:07:42.372600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.372561 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5588dd55b9-j4jbw"] Nov 02 15:07:42.374574 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.374556 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:07:42.375693 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:07:42.375660 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60b43ccc_6f07_4876_9f42_1c55eae77c3c.slice/crio-05faaf74b4bc8d4dea3eabb526ad4c9f6349c3603570128dc471066490e000f6 WatchSource:0}: Error finding container 05faaf74b4bc8d4dea3eabb526ad4c9f6349c3603570128dc471066490e000f6: Status 404 returned error can't find the container with id 05faaf74b4bc8d4dea3eabb526ad4c9f6349c3603570128dc471066490e000f6 Nov 02 15:07:42.503992 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.503940 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" event={"ID":"60b43ccc-6f07-4876-9f42-1c55eae77c3c","Type":"ContainerStarted","Data":"05faaf74b4bc8d4dea3eabb526ad4c9f6349c3603570128dc471066490e000f6"} Nov 02 15:07:42.505292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.505266 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" event={"ID":"2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0","Type":"ContainerStarted","Data":"656d5682a9c8748b032833ee181f6e8ed9742fc375fd65a5066849f4258aaf91"} Nov 02 15:07:42.506418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.506397 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" event={"ID":"010ec615-c5e2-4791-94a8-3951ce25ec1a","Type":"ContainerStarted","Data":"c5ea29abade11f85003fafc87b6354d9b5c491aba9601421c302ff16ca1e0b55"} Nov 02 15:07:42.734239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:42.734207 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b89d89487-gvj4d"] Nov 02 15:07:42.737545 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:07:42.737506 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd43b6840_ec0e_4929_bd0e_ba89875d57cb.slice/crio-66ded1cb44b627e0422ac054fb7f5588f2ca739c24ff4f4ad5a2d3ab962ed056 WatchSource:0}: Error finding container 66ded1cb44b627e0422ac054fb7f5588f2ca739c24ff4f4ad5a2d3ab962ed056: Status 404 returned error can't find the container with id 66ded1cb44b627e0422ac054fb7f5588f2ca739c24ff4f4ad5a2d3ab962ed056 Nov 02 15:07:43.511817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:43.511777 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" event={"ID":"010ec615-c5e2-4791-94a8-3951ce25ec1a","Type":"ContainerStarted","Data":"ec18a182f228ac04552badcea50a92962fa9f561f9d7f58158487610485d7256"} Nov 02 15:07:43.512304 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:43.511927 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:43.513041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:43.513016 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b89d89487-gvj4d" event={"ID":"d43b6840-ec0e-4929-bd0e-ba89875d57cb","Type":"ContainerStarted","Data":"66ded1cb44b627e0422ac054fb7f5588f2ca739c24ff4f4ad5a2d3ab962ed056"} Nov 02 15:07:43.622991 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:43.622920 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" podStartSLOduration=1.774848012 podStartE2EDuration="2.622901577s" totalImagesPullingTime="848.053565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:43.621889535 +0000 UTC m=+567.295092729" watchObservedRunningTime="2026-11-02 15:07:43.622901577 +0000 UTC m=+567.296104772" Nov 02 15:07:44.519040 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.518991 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" event={"ID":"2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0","Type":"ContainerStarted","Data":"3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd"} Nov 02 15:07:44.519040 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.519016 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" podUID="2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0" containerName="authorino" containerID="cri-o://3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd" gracePeriod=30 Nov 02 15:07:44.521194 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.521165 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b89d89487-gvj4d" event={"ID":"d43b6840-ec0e-4929-bd0e-ba89875d57cb","Type":"ContainerStarted","Data":"f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af"} Nov 02 15:07:44.523943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.523869 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" podUID="60b43ccc-6f07-4876-9f42-1c55eae77c3c" containerName="authorino" containerID="cri-o://6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc" gracePeriod=30 Nov 02 15:07:44.524104 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.524030 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" event={"ID":"60b43ccc-6f07-4876-9f42-1c55eae77c3c","Type":"ContainerStarted","Data":"6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc"} Nov 02 15:07:44.631563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.631432 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" podStartSLOduration=1.945788015 podStartE2EDuration="3.631413781s" totalImagesPullingTime="1.685625766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:44.631161178 +0000 UTC m=+568.304364374" watchObservedRunningTime="2026-11-02 15:07:44.631413781 +0000 UTC m=+568.304616976" Nov 02 15:07:44.738006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.737688 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" podStartSLOduration=2.309905033 podStartE2EDuration="3.737672526s" totalImagesPullingTime="1.427767493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:44.737274893 +0000 UTC m=+568.410478084" watchObservedRunningTime="2026-11-02 15:07:44.737672526 +0000 UTC m=+568.410875722" Nov 02 15:07:44.781595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.781537 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" Nov 02 15:07:44.817030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.817005 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" Nov 02 15:07:44.848155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.848103 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7b89d89487-gvj4d" podStartSLOduration=1.6496537820000001 podStartE2EDuration="2.848086153s" totalImagesPullingTime="1.198432371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:44.846815061 +0000 UTC m=+568.520018257" watchObservedRunningTime="2026-11-02 15:07:44.848086153 +0000 UTC m=+568.521289348" Nov 02 15:07:44.879199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.879169 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0-kube-api-access-kbvqh\" (UniqueName: \"kubernetes.io/projected/2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0-kube-api-access-kbvqh\") pod \"2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0\" (UID: \"2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0\") " Nov 02 15:07:44.879375 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.879299 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b43ccc-6f07-4876-9f42-1c55eae77c3c-kube-api-access-5xkq2\" (UniqueName: \"kubernetes.io/projected/60b43ccc-6f07-4876-9f42-1c55eae77c3c-kube-api-access-5xkq2\") pod \"60b43ccc-6f07-4876-9f42-1c55eae77c3c\" (UID: \"60b43ccc-6f07-4876-9f42-1c55eae77c3c\") " Nov 02 15:07:44.881503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.881469 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0-kube-api-access-kbvqh" pod "2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0" (UID: "2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0"). InnerVolumeSpecName "kube-api-access-kbvqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:44.881503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.881490 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b43ccc-6f07-4876-9f42-1c55eae77c3c-kube-api-access-5xkq2" pod "60b43ccc-6f07-4876-9f42-1c55eae77c3c" (UID: "60b43ccc-6f07-4876-9f42-1c55eae77c3c"). InnerVolumeSpecName "kube-api-access-5xkq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:44.980910 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.980877 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbvqh\" (UniqueName: \"kubernetes.io/projected/2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0-kube-api-access-kbvqh\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:44.980910 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:44.980906 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xkq2\" (UniqueName: \"kubernetes.io/projected/60b43ccc-6f07-4876-9f42-1c55eae77c3c-kube-api-access-5xkq2\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:45.535079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.535042 5433 generic.go:356] "Generic (PLEG): container finished" podID="60b43ccc-6f07-4876-9f42-1c55eae77c3c" containerID="6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc" exitCode=0 Nov 02 15:07:45.535566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.535084 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" Nov 02 15:07:45.535566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.535127 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" event={"ID":"60b43ccc-6f07-4876-9f42-1c55eae77c3c","Type":"ContainerDied","Data":"6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc"} Nov 02 15:07:45.535566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.535167 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5588dd55b9-j4jbw" event={"ID":"60b43ccc-6f07-4876-9f42-1c55eae77c3c","Type":"ContainerDied","Data":"05faaf74b4bc8d4dea3eabb526ad4c9f6349c3603570128dc471066490e000f6"} Nov 02 15:07:45.535566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.535188 5433 scope.go:122] "RemoveContainer" containerID="6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc" Nov 02 15:07:45.536555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.536491 5433 generic.go:356] "Generic (PLEG): container finished" podID="2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0" containerID="3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd" exitCode=0 Nov 02 15:07:45.536555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.536540 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" Nov 02 15:07:45.536653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.536563 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" event={"ID":"2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0","Type":"ContainerDied","Data":"3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd"} Nov 02 15:07:45.536653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.536592 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-h65g2" event={"ID":"2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0","Type":"ContainerDied","Data":"656d5682a9c8748b032833ee181f6e8ed9742fc375fd65a5066849f4258aaf91"} Nov 02 15:07:45.545828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.545808 5433 scope.go:122] "RemoveContainer" containerID="6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc" Nov 02 15:07:45.546166 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:07:45.546133 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc\": container with ID starting with 6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc not found: ID does not exist" containerID="6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc" Nov 02 15:07:45.546257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.546169 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc"} err="failed to get container status \"6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc\": rpc error: code = NotFound desc = could not find container \"6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc\": container with ID starting with 6f8d09a5c0ee7f9412b17e2f1748a649f511e9276884cd7b5a501a35aa0d7efc not found: ID does not exist" Nov 02 15:07:45.546257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.546196 5433 scope.go:122] "RemoveContainer" containerID="3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd" Nov 02 15:07:45.553708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.553689 5433 scope.go:122] "RemoveContainer" containerID="3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd" Nov 02 15:07:45.553953 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:07:45.553934 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd\": container with ID starting with 3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd not found: ID does not exist" containerID="3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd" Nov 02 15:07:45.554020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.553962 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd"} err="failed to get container status \"3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd\": rpc error: code = NotFound desc = could not find container \"3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd\": container with ID starting with 3bd17fed3fcdda3b01fed6003b2ac5cb4df22192df0231df40f5d1f4518976cd not found: ID does not exist" Nov 02 15:07:45.686001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.685960 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h65g2"] Nov 02 15:07:45.689861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.689838 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-h65g2"] Nov 02 15:07:45.790762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.790677 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5588dd55b9-j4jbw"] Nov 02 15:07:45.794700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:45.794672 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5588dd55b9-j4jbw"] Nov 02 15:07:46.456597 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:46.456563 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0" path="/var/lib/kubelet/pods/2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0/volumes" Nov 02 15:07:46.456880 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:46.456867 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b43ccc-6f07-4876-9f42-1c55eae77c3c" path="/var/lib/kubelet/pods/60b43ccc-6f07-4876-9f42-1c55eae77c3c/volumes" Nov 02 15:07:49.529094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:49.529066 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 15:07:49.687508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:49.687475 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj"] Nov 02 15:07:49.687804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:49.687775 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" podUID="b077df3c-3329-42c3-853e-d789ecd2cbe5" containerName="maas-api" containerID="cri-o://01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac" gracePeriod=30 Nov 02 15:07:49.927282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:49.927257 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:50.021342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.021311 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b077df3c-3329-42c3-853e-d789ecd2cbe5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b077df3c-3329-42c3-853e-d789ecd2cbe5-openshift-service-ca\") pod \"b077df3c-3329-42c3-853e-d789ecd2cbe5\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " Nov 02 15:07:50.021503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.021415 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-maas-api-tls\") pod \"b077df3c-3329-42c3-853e-d789ecd2cbe5\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " Nov 02 15:07:50.021503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.021483 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b077df3c-3329-42c3-853e-d789ecd2cbe5-kube-api-access-tfxq4\" (UniqueName: \"kubernetes.io/projected/b077df3c-3329-42c3-853e-d789ecd2cbe5-kube-api-access-tfxq4\") pod \"b077df3c-3329-42c3-853e-d789ecd2cbe5\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " Nov 02 15:07:50.021602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.021581 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-metrics-certs\") pod \"b077df3c-3329-42c3-853e-d789ecd2cbe5\" (UID: \"b077df3c-3329-42c3-853e-d789ecd2cbe5\") " Nov 02 15:07:50.021729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.021703 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b077df3c-3329-42c3-853e-d789ecd2cbe5-openshift-service-ca" pod "b077df3c-3329-42c3-853e-d789ecd2cbe5" (UID: "b077df3c-3329-42c3-853e-d789ecd2cbe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:07:50.021871 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.021856 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b077df3c-3329-42c3-853e-d789ecd2cbe5-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:50.023602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.023573 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-metrics-certs" pod "b077df3c-3329-42c3-853e-d789ecd2cbe5" (UID: "b077df3c-3329-42c3-853e-d789ecd2cbe5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:50.023709 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.023598 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-maas-api-tls" pod "b077df3c-3329-42c3-853e-d789ecd2cbe5" (UID: "b077df3c-3329-42c3-853e-d789ecd2cbe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:50.023709 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.023619 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b077df3c-3329-42c3-853e-d789ecd2cbe5-kube-api-access-tfxq4" pod "b077df3c-3329-42c3-853e-d789ecd2cbe5" (UID: "b077df3c-3329-42c3-853e-d789ecd2cbe5"). InnerVolumeSpecName "kube-api-access-tfxq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:50.122815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.122741 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:50.122815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.122767 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfxq4\" (UniqueName: \"kubernetes.io/projected/b077df3c-3329-42c3-853e-d789ecd2cbe5-kube-api-access-tfxq4\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:50.122815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.122780 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b077df3c-3329-42c3-853e-d789ecd2cbe5-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:50.560039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.560004 5433 generic.go:356] "Generic (PLEG): container finished" podID="b077df3c-3329-42c3-853e-d789ecd2cbe5" containerID="01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac" exitCode=0 Nov 02 15:07:50.560437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.560059 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" Nov 02 15:07:50.560437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.560088 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" event={"ID":"b077df3c-3329-42c3-853e-d789ecd2cbe5","Type":"ContainerDied","Data":"01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac"} Nov 02 15:07:50.560437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.560163 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj" event={"ID":"b077df3c-3329-42c3-853e-d789ecd2cbe5","Type":"ContainerDied","Data":"bc87599f8c3cced29193b8182dca98c83f29eb3e49e778bfb6d0460d09a84959"} Nov 02 15:07:50.560437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.560186 5433 scope.go:122] "RemoveContainer" containerID="01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac" Nov 02 15:07:50.568518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.568498 5433 scope.go:122] "RemoveContainer" containerID="01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac" Nov 02 15:07:50.568786 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:07:50.568764 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac\": container with ID starting with 01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac not found: ID does not exist" containerID="01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac" Nov 02 15:07:50.568886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.568794 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac"} err="failed to get container status \"01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac\": rpc error: code = NotFound desc = could not find container \"01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac\": container with ID starting with 01289e0888b41e73e66ff2a9b239b99877340be79a0655c8e5f8fa16f7672aac not found: ID does not exist" Nov 02 15:07:50.714501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.714464 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj"] Nov 02 15:07:50.760510 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:50.760424 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fprfj"] Nov 02 15:07:52.454517 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:07:52.454479 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b077df3c-3329-42c3-853e-d789ecd2cbe5" path="/var/lib/kubelet/pods/b077df3c-3329-42c3-853e-d789ecd2cbe5/volumes" Nov 02 15:08:11.010442 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:08:11.010407 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:08:19.164583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:08:19.164546 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:38.084499 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:08:38.084448 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:41.009995 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:08:41.009930 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:08:45.297142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:08:45.297086 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:08:45.436910 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:08:45.436870 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:00.215898 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:00.215824 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:00.362367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:00.362334 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:11.009893 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:09:11.009851 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:09:18.521467 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.521434 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p"] Nov 02 15:09:18.521968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.521902 5433 cpu_manager.go:414] "containerMap: removing container" podUID="2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0" containerName="authorino" Nov 02 15:09:18.521968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.521921 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0" containerName="authorino" Nov 02 15:09:18.521968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.521942 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b077df3c-3329-42c3-853e-d789ecd2cbe5" containerName="maas-api" Nov 02 15:09:18.521968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.521950 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b077df3c-3329-42c3-853e-d789ecd2cbe5" containerName="maas-api" Nov 02 15:09:18.521968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.521991 5433 cpu_manager.go:414] "containerMap: removing container" podUID="60b43ccc-6f07-4876-9f42-1c55eae77c3c" containerName="authorino" Nov 02 15:09:18.521968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.521999 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b43ccc-6f07-4876-9f42-1c55eae77c3c" containerName="authorino" Nov 02 15:09:18.522305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.522096 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b43ccc-6f07-4876-9f42-1c55eae77c3c" containerName="authorino" Nov 02 15:09:18.522305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.522115 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb47aa9-c39f-4c57-ad18-42a3acc6ffb0" containerName="authorino" Nov 02 15:09:18.522305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.522126 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b077df3c-3329-42c3-853e-d789ecd2cbe5" containerName="maas-api" Nov 02 15:09:18.524611 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.524588 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.573450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.573411 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 15:09:18.573642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.573559 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 15:09:18.573642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.573566 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 15:09:18.573767 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.573639 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-49zzg\"" Nov 02 15:09:18.610856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.610825 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:18.620405 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.620378 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p"] Nov 02 15:09:18.647920 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.647890 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e185e809-cc45-46d1-9912-e0bfc97ebb22-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.647920 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.647920 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4ph\" (UniqueName: \"kubernetes.io/projected/e185e809-cc45-46d1-9912-e0bfc97ebb22-kube-api-access-km4ph\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.648183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.647949 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.648183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.648070 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.648183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.648101 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.648183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.648141 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.748873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.748837 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.749060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.748897 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.749060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.748948 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e185e809-cc45-46d1-9912-e0bfc97ebb22-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.749060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.748993 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4ph\" (UniqueName: \"kubernetes.io/projected/e185e809-cc45-46d1-9912-e0bfc97ebb22-kube-api-access-km4ph\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.749060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.749029 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.749259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.749063 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.749353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.749331 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.749414 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.749370 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.749465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.749416 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.751224 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.751204 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e185e809-cc45-46d1-9912-e0bfc97ebb22-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.751418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.751401 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e185e809-cc45-46d1-9912-e0bfc97ebb22-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.810576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.810499 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4ph\" (UniqueName: \"kubernetes.io/projected/e185e809-cc45-46d1-9912-e0bfc97ebb22-kube-api-access-km4ph\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f9l8p\" (UID: \"e185e809-cc45-46d1-9912-e0bfc97ebb22\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:18.836425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:18.836383 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:19.200216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:19.200137 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p"] Nov 02 15:09:19.203153 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:09:19.203121 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode185e809_cc45_46d1_9912_e0bfc97ebb22.slice/crio-13239e65880f389d3ade5225728493bb40eab02738e1a41a5bddb2e0bfbe20ca WatchSource:0}: Error finding container 13239e65880f389d3ade5225728493bb40eab02738e1a41a5bddb2e0bfbe20ca: Status 404 returned error can't find the container with id 13239e65880f389d3ade5225728493bb40eab02738e1a41a5bddb2e0bfbe20ca Nov 02 15:09:19.484335 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:19.484241 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:19.918533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:19.918494 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" event={"ID":"e185e809-cc45-46d1-9912-e0bfc97ebb22","Type":"ContainerStarted","Data":"13239e65880f389d3ade5225728493bb40eab02738e1a41a5bddb2e0bfbe20ca"} Nov 02 15:09:21.928527 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:21.928488 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" event={"ID":"e185e809-cc45-46d1-9912-e0bfc97ebb22","Type":"ContainerStarted","Data":"05f01355d2d571d48aef242b23761065cb20de0dcda4eddb63a8445cf5dcb254"} Nov 02 15:09:22.040196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:22.040123 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" podStartSLOduration=2.096670945 podStartE2EDuration="4.040102873s" totalImagesPullingTime="1.943431928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:22.039287462 +0000 UTC m=+665.712490657" watchObservedRunningTime="2026-11-02 15:09:22.040102873 +0000 UTC m=+665.713306072" Nov 02 15:09:22.837035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:22.837000 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:24.984917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:24.984875 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5"] Nov 02 15:09:24.987634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:24.987616 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.077388 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.077351 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:25.082049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.082022 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 15:09:25.083077 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.083054 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5"] Nov 02 15:09:25.105869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.105842 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.106019 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.105884 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae29092-e319-49bd-ae57-741c217a1e03-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.106019 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.105905 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.106123 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.106049 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcj7j\" (UniqueName: \"kubernetes.io/projected/3ae29092-e319-49bd-ae57-741c217a1e03-kube-api-access-pcj7j\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.106123 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.106078 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.106123 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.106111 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.206795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.206759 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcj7j\" (UniqueName: \"kubernetes.io/projected/3ae29092-e319-49bd-ae57-741c217a1e03-kube-api-access-pcj7j\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.207024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.206801 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.207024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.206833 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.207024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.206870 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.207024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.206897 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae29092-e319-49bd-ae57-741c217a1e03-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.207024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.206915 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.207336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.207315 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.207386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.207363 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.207449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.207427 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.209133 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.209114 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae29092-e319-49bd-ae57-741c217a1e03-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.209453 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.209435 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae29092-e319-49bd-ae57-741c217a1e03-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.263898 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.263830 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcj7j\" (UniqueName: \"kubernetes.io/projected/3ae29092-e319-49bd-ae57-741c217a1e03-kube-api-access-pcj7j\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-85zn5\" (UID: \"3ae29092-e319-49bd-ae57-741c217a1e03\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.299695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.299662 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:25.662336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.662304 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5"] Nov 02 15:09:25.666032 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:09:25.665999 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae29092_e319_49bd_ae57_741c217a1e03.slice/crio-a07db2b9b41af51a2f9da5dd352b144d9ec6d2def04501d5c2db9238352885a8 WatchSource:0}: Error finding container a07db2b9b41af51a2f9da5dd352b144d9ec6d2def04501d5c2db9238352885a8: Status 404 returned error can't find the container with id a07db2b9b41af51a2f9da5dd352b144d9ec6d2def04501d5c2db9238352885a8 Nov 02 15:09:25.667503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.667484 5433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:09:25.952427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:25.952348 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" event={"ID":"3ae29092-e319-49bd-ae57-741c217a1e03","Type":"ContainerStarted","Data":"a07db2b9b41af51a2f9da5dd352b144d9ec6d2def04501d5c2db9238352885a8"} Nov 02 15:09:26.958284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:26.958251 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" event={"ID":"3ae29092-e319-49bd-ae57-741c217a1e03","Type":"ContainerStarted","Data":"1acb3d05c1d83ab83f3dbef30aee6bb8e3687b5863860ac7d0daccf5de4f896f"} Nov 02 15:09:27.070111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:27.070047 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" podStartSLOduration=2.444897856 podStartE2EDuration="3.070026756s" totalImagesPullingTime="625.1289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:27.068917325 +0000 UTC m=+670.742120503" watchObservedRunningTime="2026-11-02 15:09:27.070026756 +0000 UTC m=+670.743229954" Nov 02 15:09:27.300493 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:27.300409 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:28.231467 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.231431 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr"] Nov 02 15:09:28.234392 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.234375 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.319272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.319240 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:28.328852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.328826 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 15:09:28.330316 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.330288 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr"] Nov 02 15:09:28.332246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.332221 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjl9s\" (UniqueName: \"kubernetes.io/projected/65866332-3ec1-4250-8f95-67bcf624745d-kube-api-access-bjl9s\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.332370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.332280 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65866332-3ec1-4250-8f95-67bcf624745d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.332424 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.332373 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.332424 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.332408 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.332491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.332424 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.332491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.332466 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.433674 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.433637 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65866332-3ec1-4250-8f95-67bcf624745d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.433882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.433709 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.433882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.433742 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.433882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.433762 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.433882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.433787 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.433882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.433830 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjl9s\" (UniqueName: \"kubernetes.io/projected/65866332-3ec1-4250-8f95-67bcf624745d-kube-api-access-bjl9s\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.434290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.434265 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.434357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.434295 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.434357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.434315 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.436038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.436018 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65866332-3ec1-4250-8f95-67bcf624745d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.436222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.436203 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65866332-3ec1-4250-8f95-67bcf624745d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.491619 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.491554 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjl9s\" (UniqueName: \"kubernetes.io/projected/65866332-3ec1-4250-8f95-67bcf624745d-kube-api-access-bjl9s\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4p5hr\" (UID: \"65866332-3ec1-4250-8f95-67bcf624745d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.546665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.546634 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:28.837085 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.837005 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:28.850087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.850063 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:28.910950 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.910928 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr"] Nov 02 15:09:28.914694 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:09:28.914661 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65866332_3ec1_4250_8f95_67bcf624745d.slice/crio-e4c61ca0fa926445ff4edbde367479da158f9ddcab5ea6fac8651d57f48b3fe7 WatchSource:0}: Error finding container e4c61ca0fa926445ff4edbde367479da158f9ddcab5ea6fac8651d57f48b3fe7: Status 404 returned error can't find the container with id e4c61ca0fa926445ff4edbde367479da158f9ddcab5ea6fac8651d57f48b3fe7 Nov 02 15:09:28.967219 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.967188 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" event={"ID":"65866332-3ec1-4250-8f95-67bcf624745d","Type":"ContainerStarted","Data":"e4c61ca0fa926445ff4edbde367479da158f9ddcab5ea6fac8651d57f48b3fe7"} Nov 02 15:09:28.978534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:28.978518 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f9l8p" Nov 02 15:09:29.973790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:29.973749 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" event={"ID":"65866332-3ec1-4250-8f95-67bcf624745d","Type":"ContainerStarted","Data":"0e38694c3c4744c95eb1ec7baa5a4af1e6ec8376841bcbd23fda8bff0dde17cf"} Nov 02 15:09:30.084085 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:30.084023 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" podStartSLOduration=1.419677937 podStartE2EDuration="2.084006205s" totalImagesPullingTime="664.328268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:30.083613699 +0000 UTC m=+673.756816896" watchObservedRunningTime="2026-11-02 15:09:30.084006205 +0000 UTC m=+673.757209397" Nov 02 15:09:30.547401 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:30.547364 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:31.433774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:31.433741 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:34.905075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:34.905048 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:35.300968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:35.300889 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:35.313411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:35.313388 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:36.016744 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:36.016710 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-85zn5" Nov 02 15:09:38.547531 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:38.547493 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:38.560144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:38.560110 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:38.962710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:38.962672 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5"] Nov 02 15:09:38.965564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:38.965547 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.014020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.013961 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 15:09:39.033112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.033081 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4p5hr" Nov 02 15:09:39.035597 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.035567 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dft\" (UniqueName: \"kubernetes.io/projected/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-kube-api-access-96dft\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.035678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.035615 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.035756 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.035742 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.035789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.035774 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.035882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.035868 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.035916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.035897 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.052715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.052688 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:39.061921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.061894 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5"] Nov 02 15:09:39.137290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.137260 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dft\" (UniqueName: \"kubernetes.io/projected/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-kube-api-access-96dft\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.137478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.137299 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.137478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.137343 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.137478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.137363 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.137478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.137425 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.137478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.137451 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.137760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.137718 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.137815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.137801 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.137859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.137820 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.139684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.139654 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.140057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.140035 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.195591 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.195527 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dft\" (UniqueName: \"kubernetes.io/projected/89a646d3-ed9b-4cff-9d54-1ba94e3bfc00-kube-api-access-96dft\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5\" (UID: \"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.278303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.278219 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:39.641409 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:39.641377 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5"] Nov 02 15:09:39.644900 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:09:39.644861 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a646d3_ed9b_4cff_9d54_1ba94e3bfc00.slice/crio-a5a19c66091cabbef080739b8938f69c2d1295f20a2eb05d24c84837accf3af7 WatchSource:0}: Error finding container a5a19c66091cabbef080739b8938f69c2d1295f20a2eb05d24c84837accf3af7: Status 404 returned error can't find the container with id a5a19c66091cabbef080739b8938f69c2d1295f20a2eb05d24c84837accf3af7 Nov 02 15:09:40.025339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:40.025250 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" event={"ID":"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00","Type":"ContainerStarted","Data":"a5a19c66091cabbef080739b8938f69c2d1295f20a2eb05d24c84837accf3af7"} Nov 02 15:09:41.010481 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:09:41.010444 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:09:41.032725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:41.032694 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" event={"ID":"89a646d3-ed9b-4cff-9d54-1ba94e3bfc00","Type":"ContainerStarted","Data":"8a356445b8d3e47d23bcabf32d546e5c4eae5db88da78bed710bab7d981af423"} Nov 02 15:09:41.142651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:41.142589 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" podStartSLOduration=2.51431715 podStartE2EDuration="3.142572763s" totalImagesPullingTime="628.255613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:41.142040428 +0000 UTC m=+684.815243623" watchObservedRunningTime="2026-11-02 15:09:41.142572763 +0000 UTC m=+684.815775958" Nov 02 15:09:41.279083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:41.279012 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:42.428875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:42.428830 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:45.522699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.522662 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88"] Nov 02 15:09:45.526416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.526398 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.596459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.596418 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 15:09:45.601752 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.601723 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.601889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.601771 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.602035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.602013 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.602098 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.602062 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccbwf\" (UniqueName: \"kubernetes.io/projected/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-kube-api-access-ccbwf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.602136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.602130 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.602381 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.602362 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.611427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.611406 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:09:45.622109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.622079 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88"] Nov 02 15:09:45.703680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.703640 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.703840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.703709 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.703840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.703750 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.703840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.703782 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.703840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.703813 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.703840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.703829 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccbwf\" (UniqueName: \"kubernetes.io/projected/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-kube-api-access-ccbwf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.704214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.704187 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.704317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.704196 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.704317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.704288 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.705888 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.705867 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.706216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.706200 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.760647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.760614 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccbwf\" (UniqueName: \"kubernetes.io/projected/9590d5ee-9ed8-4f51-bd39-b044c4a7f80f-kube-api-access-ccbwf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88\" (UID: \"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:45.838217 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:45.838130 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:46.199852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:46.199756 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88"] Nov 02 15:09:46.202640 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:09:46.202605 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9590d5ee_9ed8_4f51_bd39_b044c4a7f80f.slice/crio-84ef42a27b9347144ed4971c781936eb88f42cd5bb55cd427f9b8c877cc256c4 WatchSource:0}: Error finding container 84ef42a27b9347144ed4971c781936eb88f42cd5bb55cd427f9b8c877cc256c4: Status 404 returned error can't find the container with id 84ef42a27b9347144ed4971c781936eb88f42cd5bb55cd427f9b8c877cc256c4 Nov 02 15:09:47.083720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:47.083629 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" event={"ID":"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f","Type":"ContainerStarted","Data":"4c8539254a94a9a874a83e679afd15090263fa4cd04b6c56368059e9889dac6e"} Nov 02 15:09:47.083720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:47.083692 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" event={"ID":"9590d5ee-9ed8-4f51-bd39-b044c4a7f80f","Type":"ContainerStarted","Data":"84ef42a27b9347144ed4971c781936eb88f42cd5bb55cd427f9b8c877cc256c4"} Nov 02 15:09:47.193687 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:47.193636 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" podStartSLOduration=1.57406543 podStartE2EDuration="2.193620865s" totalImagesPullingTime="619.555435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:47.193422352 +0000 UTC m=+690.866625548" watchObservedRunningTime="2026-11-02 15:09:47.193620865 +0000 UTC m=+690.866824059" Nov 02 15:09:47.839020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:47.838986 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:48.270424 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.270386 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb"] Nov 02 15:09:48.273114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.273095 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.326792 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.326754 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x6s8k\"" Nov 02 15:09:48.373721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.373688 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb"] Nov 02 15:09:48.425897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.425861 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.425897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.425903 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.426136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.426034 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.426136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.426058 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da392a7d-e24e-4dd1-b5f0-991320a94769-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.426136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.426079 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.426136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.426098 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.426265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.426204 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbfn7\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-kube-api-access-zbfn7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.426265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.426249 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.426328 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.426300 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.527821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.527737 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.527821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.527780 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da392a7d-e24e-4dd1-b5f0-991320a94769-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.527821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.527801 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.527824 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.527858 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbfn7\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-kube-api-access-zbfn7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.527902 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.527929 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.528018 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.528050 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.528311 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528471 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.528402 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528471 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.528457 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528587 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.528563 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.528670 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.528654 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da392a7d-e24e-4dd1-b5f0-991320a94769-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.530179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.530154 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.530388 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.530369 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.585922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.585889 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.586134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.586083 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbfn7\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-kube-api-access-zbfn7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s6jxb\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:48.885510 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:48.885475 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:49.279383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:49.279282 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:49.292645 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:49.292619 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:49.348158 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:09:49.348128 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda392a7d_e24e_4dd1_b5f0_991320a94769.slice/crio-29ecb082e4f843b0690168947a9a7b2afb6c5f42fe049a3c37833f647fd55fb6 WatchSource:0}: Error finding container 29ecb082e4f843b0690168947a9a7b2afb6c5f42fe049a3c37833f647fd55fb6: Status 404 returned error can't find the container with id 29ecb082e4f843b0690168947a9a7b2afb6c5f42fe049a3c37833f647fd55fb6 Nov 02 15:09:49.390418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:49.390378 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb"] Nov 02 15:09:50.099351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:50.099309 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" event={"ID":"da392a7d-e24e-4dd1-b5f0-991320a94769","Type":"ContainerStarted","Data":"29ecb082e4f843b0690168947a9a7b2afb6c5f42fe049a3c37833f647fd55fb6"} Nov 02 15:09:50.110899 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:50.110868 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5" Nov 02 15:09:53.914561 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:53.914530 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:09:53.914822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:53.914601 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:09:53.914822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:53.914631 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:09:54.120216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:54.120177 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" event={"ID":"da392a7d-e24e-4dd1-b5f0-991320a94769","Type":"ContainerStarted","Data":"9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9"} Nov 02 15:09:54.281276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:54.281160 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podStartSLOduration=1.717669158 podStartE2EDuration="6.281140949s" totalImagesPullingTime="4.563471791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:54.279106155 +0000 UTC m=+697.952309350" watchObservedRunningTime="2026-11-02 15:09:54.281140949 +0000 UTC m=+697.954344144" Nov 02 15:09:54.885877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:54.885836 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:09:54.887404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:54.887382 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s6jxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 15:09:54.887526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:54.887425 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 15:09:55.841119 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:55.841067 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:55.853726 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:55.853697 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:55.886312 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:55.886286 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s6jxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 15:09:55.886453 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:55.886334 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 15:09:56.156550 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:56.156522 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88" Nov 02 15:09:56.886809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:56.886771 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s6jxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 15:09:56.887256 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:56.886828 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 15:09:57.885911 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:57.885877 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s6jxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 15:09:57.886113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:57.885931 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 15:09:58.885795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:58.885761 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s6jxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 15:09:58.886194 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:58.885823 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 15:09:59.886645 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:59.886601 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s6jxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 15:09:59.887037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:09:59.886673 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 15:10:00.886433 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:00.886403 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s6jxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 15:10:00.886584 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:00.886481 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 15:10:02.136100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:02.136064 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:10:02.136553 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:02.136411 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:10:02.137603 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:02.137582 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:10:11.017989 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:10:11.017931 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:10:11.112835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.112786 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:11.132053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.132020 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-tnm4n"] Nov 02 15:10:11.135531 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.135509 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:11.150302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.150278 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e421f0ac-f84e-463e-9c8a-990e37270e02-metrics-certs\") pod \"ai-gateway-controller-5d46d8f988-tnm4n\" (UID: \"e421f0ac-f84e-463e-9c8a-990e37270e02\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:11.150427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.150410 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvbgf\" (UniqueName: \"kubernetes.io/projected/e421f0ac-f84e-463e-9c8a-990e37270e02-kube-api-access-wvbgf\") pod \"ai-gateway-controller-5d46d8f988-tnm4n\" (UID: \"e421f0ac-f84e-463e-9c8a-990e37270e02\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:11.230101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.230070 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 15:10:11.230101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.230094 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 15:10:11.230348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.230111 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-bgzqj\"" Nov 02 15:10:11.231369 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.231350 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-tnm4n"] Nov 02 15:10:11.251613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.251590 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbgf\" (UniqueName: \"kubernetes.io/projected/e421f0ac-f84e-463e-9c8a-990e37270e02-kube-api-access-wvbgf\") pod \"ai-gateway-controller-5d46d8f988-tnm4n\" (UID: \"e421f0ac-f84e-463e-9c8a-990e37270e02\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:11.251741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.251644 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e421f0ac-f84e-463e-9c8a-990e37270e02-metrics-certs\") pod \"ai-gateway-controller-5d46d8f988-tnm4n\" (UID: \"e421f0ac-f84e-463e-9c8a-990e37270e02\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:11.254024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.254004 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e421f0ac-f84e-463e-9c8a-990e37270e02-metrics-certs\") pod \"ai-gateway-controller-5d46d8f988-tnm4n\" (UID: \"e421f0ac-f84e-463e-9c8a-990e37270e02\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:11.271713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.271640 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:11.309386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.309353 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbgf\" (UniqueName: \"kubernetes.io/projected/e421f0ac-f84e-463e-9c8a-990e37270e02-kube-api-access-wvbgf\") pod \"ai-gateway-controller-5d46d8f988-tnm4n\" (UID: \"e421f0ac-f84e-463e-9c8a-990e37270e02\") " pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:11.448100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.448074 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:11.811073 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:11.811033 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-tnm4n"] Nov 02 15:10:11.814623 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:10:11.814586 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode421f0ac_f84e_463e_9c8a_990e37270e02.slice/crio-b3950d7709064d863a18140fd8f0212c035f104d8be9855557c6a85e716ddbb3 WatchSource:0}: Error finding container b3950d7709064d863a18140fd8f0212c035f104d8be9855557c6a85e716ddbb3: Status 404 returned error can't find the container with id b3950d7709064d863a18140fd8f0212c035f104d8be9855557c6a85e716ddbb3 Nov 02 15:10:12.217339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:12.217305 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" event={"ID":"e421f0ac-f84e-463e-9c8a-990e37270e02","Type":"ContainerStarted","Data":"b3950d7709064d863a18140fd8f0212c035f104d8be9855557c6a85e716ddbb3"} Nov 02 15:10:14.179503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:14.179461 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:15.320569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:15.320532 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b89d89487-gvj4d"] Nov 02 15:10:15.321074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:15.320792 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7b89d89487-gvj4d" podUID="d43b6840-ec0e-4929-bd0e-ba89875d57cb" containerName="authorino" containerID="cri-o://f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af" gracePeriod=30 Nov 02 15:10:15.574537 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:15.574479 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:10:15.596719 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:15.596693 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43b6840-ec0e-4929-bd0e-ba89875d57cb-kube-api-access-jsvv4\" (UniqueName: \"kubernetes.io/projected/d43b6840-ec0e-4929-bd0e-ba89875d57cb-kube-api-access-jsvv4\") pod \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\" (UID: \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\") " Nov 02 15:10:15.596819 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:15.596749 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43b6840-ec0e-4929-bd0e-ba89875d57cb-tls-cert\" (UniqueName: \"kubernetes.io/secret/d43b6840-ec0e-4929-bd0e-ba89875d57cb-tls-cert\") pod \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\" (UID: \"d43b6840-ec0e-4929-bd0e-ba89875d57cb\") " Nov 02 15:10:15.598754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:15.598728 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43b6840-ec0e-4929-bd0e-ba89875d57cb-kube-api-access-jsvv4" pod "d43b6840-ec0e-4929-bd0e-ba89875d57cb" (UID: "d43b6840-ec0e-4929-bd0e-ba89875d57cb"). InnerVolumeSpecName "kube-api-access-jsvv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:15.606970 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:15.606946 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43b6840-ec0e-4929-bd0e-ba89875d57cb-tls-cert" pod "d43b6840-ec0e-4929-bd0e-ba89875d57cb" (UID: "d43b6840-ec0e-4929-bd0e-ba89875d57cb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:10:15.698263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:15.698226 5433 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d43b6840-ec0e-4929-bd0e-ba89875d57cb-tls-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:10:15.698263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:15.698259 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsvv4\" (UniqueName: \"kubernetes.io/projected/d43b6840-ec0e-4929-bd0e-ba89875d57cb-kube-api-access-jsvv4\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:10:16.238470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.238433 5433 generic.go:356] "Generic (PLEG): container finished" podID="d43b6840-ec0e-4929-bd0e-ba89875d57cb" containerID="f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af" exitCode=0 Nov 02 15:10:16.238655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.238480 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b89d89487-gvj4d" Nov 02 15:10:16.238655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.238485 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b89d89487-gvj4d" event={"ID":"d43b6840-ec0e-4929-bd0e-ba89875d57cb","Type":"ContainerDied","Data":"f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af"} Nov 02 15:10:16.238655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.238533 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b89d89487-gvj4d" event={"ID":"d43b6840-ec0e-4929-bd0e-ba89875d57cb","Type":"ContainerDied","Data":"66ded1cb44b627e0422ac054fb7f5588f2ca739c24ff4f4ad5a2d3ab962ed056"} Nov 02 15:10:16.238655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.238554 5433 scope.go:122] "RemoveContainer" containerID="f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af" Nov 02 15:10:16.247796 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.247777 5433 scope.go:122] "RemoveContainer" containerID="f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af" Nov 02 15:10:16.248063 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:10:16.248044 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af\": container with ID starting with f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af not found: ID does not exist" containerID="f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af" Nov 02 15:10:16.248132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.248071 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af"} err="failed to get container status \"f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af\": rpc error: code = NotFound desc = could not find container \"f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af\": container with ID starting with f50ed04067e19faf1b4c6de3365a550ac479d3b4d1ea72d006301f33a60868af not found: ID does not exist" Nov 02 15:10:16.390580 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.390541 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b89d89487-gvj4d"] Nov 02 15:10:16.436370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.436337 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b89d89487-gvj4d"] Nov 02 15:10:16.455207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:16.455163 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43b6840-ec0e-4929-bd0e-ba89875d57cb" path="/var/lib/kubelet/pods/d43b6840-ec0e-4929-bd0e-ba89875d57cb/volumes" Nov 02 15:10:18.377919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:18.377888 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 15:10:19.253295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:19.253252 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" event={"ID":"e421f0ac-f84e-463e-9c8a-990e37270e02","Type":"ContainerStarted","Data":"da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b"} Nov 02 15:10:19.253473 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:19.253390 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:19.409576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:19.409513 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" podStartSLOduration=1.8960162459999998 podStartE2EDuration="8.409500628s" totalImagesPullingTime="6.513484382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:19.363375387 +0000 UTC m=+723.036578581" watchObservedRunningTime="2026-11-02 15:10:19.409500628 +0000 UTC m=+723.082703823" Nov 02 15:10:19.810657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:19.810614 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:21.334727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.334683 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:21.518491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.518454 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:21.753670 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.753629 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj"] Nov 02 15:10:21.754058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.754044 5433 cpu_manager.go:414] "containerMap: removing container" podUID="d43b6840-ec0e-4929-bd0e-ba89875d57cb" containerName="authorino" Nov 02 15:10:21.754112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.754060 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43b6840-ec0e-4929-bd0e-ba89875d57cb" containerName="authorino" Nov 02 15:10:21.754148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.754128 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43b6840-ec0e-4929-bd0e-ba89875d57cb" containerName="authorino" Nov 02 15:10:21.757442 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.757412 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:21.805896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.805869 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 15:10:21.852075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.852034 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-t2r5j\"" Nov 02 15:10:21.853403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.853378 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj"] Nov 02 15:10:21.855398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.855364 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9v6\" (UniqueName: \"kubernetes.io/projected/9c31e312-7bec-41e3-84b0-a6279ed4ad74-kube-api-access-gh9v6\") pod \"payload-pre-processing-7b98bb678c-6tbpj\" (UID: \"9c31e312-7bec-41e3-84b0-a6279ed4ad74\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:21.855511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.855486 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c31e312-7bec-41e3-84b0-a6279ed4ad74-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6tbpj\" (UID: \"9c31e312-7bec-41e3-84b0-a6279ed4ad74\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:21.896718 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.896685 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z9f75"] Nov 02 15:10:21.900419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.900396 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:21.956357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.956312 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z9f75"] Nov 02 15:10:21.956734 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.956690 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9v6\" (UniqueName: \"kubernetes.io/projected/9c31e312-7bec-41e3-84b0-a6279ed4ad74-kube-api-access-gh9v6\") pod \"payload-pre-processing-7b98bb678c-6tbpj\" (UID: \"9c31e312-7bec-41e3-84b0-a6279ed4ad74\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:21.956858 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.956795 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c31e312-7bec-41e3-84b0-a6279ed4ad74-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6tbpj\" (UID: \"9c31e312-7bec-41e3-84b0-a6279ed4ad74\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:21.956858 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.956844 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-z9f75\" (UID: \"e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:21.956956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.956875 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77f9\" (UniqueName: \"kubernetes.io/projected/e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f-kube-api-access-q77f9\") pod \"payload-processing-55fbbbf9dc-z9f75\" (UID: \"e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:21.957492 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:21.957468 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c31e312-7bec-41e3-84b0-a6279ed4ad74-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6tbpj\" (UID: \"9c31e312-7bec-41e3-84b0-a6279ed4ad74\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:22.014845 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:22.014770 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9v6\" (UniqueName: \"kubernetes.io/projected/9c31e312-7bec-41e3-84b0-a6279ed4ad74-kube-api-access-gh9v6\") pod \"payload-pre-processing-7b98bb678c-6tbpj\" (UID: \"9c31e312-7bec-41e3-84b0-a6279ed4ad74\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:22.057406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:22.057365 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-z9f75\" (UID: \"e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:22.057406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:22.057414 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77f9\" (UniqueName: \"kubernetes.io/projected/e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f-kube-api-access-q77f9\") pod \"payload-processing-55fbbbf9dc-z9f75\" (UID: \"e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:22.058271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:22.058243 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-z9f75\" (UID: \"e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:22.070281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:22.070256 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:22.115441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:22.115407 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77f9\" (UniqueName: \"kubernetes.io/projected/e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f-kube-api-access-q77f9\") pod \"payload-processing-55fbbbf9dc-z9f75\" (UID: \"e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:22.213904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:22.213862 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:22.440383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:22.440354 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj"] Nov 02 15:10:22.443087 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:10:22.443052 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c31e312_7bec_41e3_84b0_a6279ed4ad74.slice/crio-779b0208678b734661a5f1fcd31dde8813cc8e690e1c964348addc0b77c4d7c1 WatchSource:0}: Error finding container 779b0208678b734661a5f1fcd31dde8813cc8e690e1c964348addc0b77c4d7c1: Status 404 returned error can't find the container with id 779b0208678b734661a5f1fcd31dde8813cc8e690e1c964348addc0b77c4d7c1 Nov 02 15:10:22.581872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:22.581818 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z9f75"] Nov 02 15:10:22.584632 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:10:22.584603 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode276f1d0_b6e5_4dba_8883_a5bfc3a6c77f.slice/crio-e3f25364371c268b954ca635e589aa8599fba6771705a5039409d0b436bb3a72 WatchSource:0}: Error finding container e3f25364371c268b954ca635e589aa8599fba6771705a5039409d0b436bb3a72: Status 404 returned error can't find the container with id e3f25364371c268b954ca635e589aa8599fba6771705a5039409d0b436bb3a72 Nov 02 15:10:23.276246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:23.276207 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" event={"ID":"e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f","Type":"ContainerStarted","Data":"e3f25364371c268b954ca635e589aa8599fba6771705a5039409d0b436bb3a72"} Nov 02 15:10:23.277752 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:23.277719 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" event={"ID":"9c31e312-7bec-41e3-84b0-a6279ed4ad74","Type":"ContainerStarted","Data":"779b0208678b734661a5f1fcd31dde8813cc8e690e1c964348addc0b77c4d7c1"} Nov 02 15:10:26.294027 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:26.293930 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" event={"ID":"e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f","Type":"ContainerStarted","Data":"e0effe2dfa719636d7fac5826aaaa0b0868f70fbad6a894a6d9de8d2bf779321"} Nov 02 15:10:26.294462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:26.294047 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:26.295546 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:26.295523 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" event={"ID":"9c31e312-7bec-41e3-84b0-a6279ed4ad74","Type":"ContainerStarted","Data":"4816d70f50136089bbc88f9ccaa0d0d17d1c599f02ea667817bdb1621695c85a"} Nov 02 15:10:26.295663 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:26.295581 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:26.404188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:26.404136 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" podStartSLOduration=2.466372288 podStartE2EDuration="5.404121128s" totalImagesPullingTime="2.93774884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:26.402846637 +0000 UTC m=+730.076049831" watchObservedRunningTime="2026-11-02 15:10:26.404121128 +0000 UTC m=+730.077324341" Nov 02 15:10:26.511491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:26.511432 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" podStartSLOduration=2.278159456 podStartE2EDuration="5.511416709s" totalImagesPullingTime="3.233257253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:26.511233666 +0000 UTC m=+730.184436861" watchObservedRunningTime="2026-11-02 15:10:26.511416709 +0000 UTC m=+730.184619902" Nov 02 15:10:30.261064 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.261029 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:30.612464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.612382 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-84664687bf-sqhrz"] Nov 02 15:10:30.615302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.615284 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:30.671733 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.671696 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84664687bf-sqhrz"] Nov 02 15:10:30.739302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.739265 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5680c5f2-72f0-411b-b2cd-baaae0ba3807-metrics-certs\") pod \"ai-gateway-controller-84664687bf-sqhrz\" (UID: \"5680c5f2-72f0-411b-b2cd-baaae0ba3807\") " pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:30.739480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.739434 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj286\" (UniqueName: \"kubernetes.io/projected/5680c5f2-72f0-411b-b2cd-baaae0ba3807-kube-api-access-bj286\") pod \"ai-gateway-controller-84664687bf-sqhrz\" (UID: \"5680c5f2-72f0-411b-b2cd-baaae0ba3807\") " pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:30.840701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.840666 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5680c5f2-72f0-411b-b2cd-baaae0ba3807-metrics-certs\") pod \"ai-gateway-controller-84664687bf-sqhrz\" (UID: \"5680c5f2-72f0-411b-b2cd-baaae0ba3807\") " pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:30.840901 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.840775 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj286\" (UniqueName: \"kubernetes.io/projected/5680c5f2-72f0-411b-b2cd-baaae0ba3807-kube-api-access-bj286\") pod \"ai-gateway-controller-84664687bf-sqhrz\" (UID: \"5680c5f2-72f0-411b-b2cd-baaae0ba3807\") " pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:30.843120 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.843096 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5680c5f2-72f0-411b-b2cd-baaae0ba3807-metrics-certs\") pod \"ai-gateway-controller-84664687bf-sqhrz\" (UID: \"5680c5f2-72f0-411b-b2cd-baaae0ba3807\") " pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:30.898559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.898525 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj286\" (UniqueName: \"kubernetes.io/projected/5680c5f2-72f0-411b-b2cd-baaae0ba3807-kube-api-access-bj286\") pod \"ai-gateway-controller-84664687bf-sqhrz\" (UID: \"5680c5f2-72f0-411b-b2cd-baaae0ba3807\") " pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:30.927306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:30.927276 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:31.290645 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:31.290558 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84664687bf-sqhrz"] Nov 02 15:10:31.293612 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:10:31.293577 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5680c5f2_72f0_411b_b2cd_baaae0ba3807.slice/crio-cacc9dbdca42fde0a89da577524ffdc87bc8ef153336d6dd2047c79252e0bfae WatchSource:0}: Error finding container cacc9dbdca42fde0a89da577524ffdc87bc8ef153336d6dd2047c79252e0bfae: Status 404 returned error can't find the container with id cacc9dbdca42fde0a89da577524ffdc87bc8ef153336d6dd2047c79252e0bfae Nov 02 15:10:31.320535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:31.320502 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" event={"ID":"5680c5f2-72f0-411b-b2cd-baaae0ba3807","Type":"ContainerStarted","Data":"cacc9dbdca42fde0a89da577524ffdc87bc8ef153336d6dd2047c79252e0bfae"} Nov 02 15:10:32.327128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:32.327083 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" event={"ID":"5680c5f2-72f0-411b-b2cd-baaae0ba3807","Type":"ContainerStarted","Data":"3b15dacc9ced71d57815db84852cea036f3274fdbf9d55f3f6a431c5a0ac39ba"} Nov 02 15:10:32.327614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:32.327276 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:32.451998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:32.451876 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" podStartSLOduration=1.619362385 podStartE2EDuration="2.451861943s" totalImagesPullingTime="832.499558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:32.449487566 +0000 UTC m=+736.122690760" watchObservedRunningTime="2026-11-02 15:10:32.451861943 +0000 UTC m=+736.125065138" Nov 02 15:10:37.303386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:37.303351 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-z9f75" Nov 02 15:10:37.303770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:37.303422 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tbpj" Nov 02 15:10:41.009352 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:10:41.009312 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:10:43.333368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.333332 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-84664687bf-sqhrz" Nov 02 15:10:43.490649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.490616 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-tnm4n"] Nov 02 15:10:43.490892 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.490859 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" podUID="e421f0ac-f84e-463e-9c8a-990e37270e02" containerName="manager" containerID="cri-o://da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b" gracePeriod=30 Nov 02 15:10:43.741055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.741030 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:43.862495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.862422 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e421f0ac-f84e-463e-9c8a-990e37270e02-kube-api-access-wvbgf\" (UniqueName: \"kubernetes.io/projected/e421f0ac-f84e-463e-9c8a-990e37270e02-kube-api-access-wvbgf\") pod \"e421f0ac-f84e-463e-9c8a-990e37270e02\" (UID: \"e421f0ac-f84e-463e-9c8a-990e37270e02\") " Nov 02 15:10:43.862495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.862473 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e421f0ac-f84e-463e-9c8a-990e37270e02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e421f0ac-f84e-463e-9c8a-990e37270e02-metrics-certs\") pod \"e421f0ac-f84e-463e-9c8a-990e37270e02\" (UID: \"e421f0ac-f84e-463e-9c8a-990e37270e02\") " Nov 02 15:10:43.864522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.864483 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e421f0ac-f84e-463e-9c8a-990e37270e02-kube-api-access-wvbgf" pod "e421f0ac-f84e-463e-9c8a-990e37270e02" (UID: "e421f0ac-f84e-463e-9c8a-990e37270e02"). InnerVolumeSpecName "kube-api-access-wvbgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:43.864644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.864539 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e421f0ac-f84e-463e-9c8a-990e37270e02-metrics-certs" pod "e421f0ac-f84e-463e-9c8a-990e37270e02" (UID: "e421f0ac-f84e-463e-9c8a-990e37270e02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:10:43.963694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.963662 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvbgf\" (UniqueName: \"kubernetes.io/projected/e421f0ac-f84e-463e-9c8a-990e37270e02-kube-api-access-wvbgf\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:10:43.963694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:43.963690 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e421f0ac-f84e-463e-9c8a-990e37270e02-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:10:44.379067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:44.379033 5433 generic.go:356] "Generic (PLEG): container finished" podID="e421f0ac-f84e-463e-9c8a-990e37270e02" containerID="da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b" exitCode=0 Nov 02 15:10:44.379548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:44.379090 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" Nov 02 15:10:44.379548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:44.379117 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" event={"ID":"e421f0ac-f84e-463e-9c8a-990e37270e02","Type":"ContainerDied","Data":"da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b"} Nov 02 15:10:44.379548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:44.379215 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d46d8f988-tnm4n" event={"ID":"e421f0ac-f84e-463e-9c8a-990e37270e02","Type":"ContainerDied","Data":"b3950d7709064d863a18140fd8f0212c035f104d8be9855557c6a85e716ddbb3"} Nov 02 15:10:44.379548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:44.379249 5433 scope.go:122] "RemoveContainer" containerID="da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b" Nov 02 15:10:44.388460 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:44.388443 5433 scope.go:122] "RemoveContainer" containerID="da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b" Nov 02 15:10:44.388725 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:10:44.388706 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b\": container with ID starting with da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b not found: ID does not exist" containerID="da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b" Nov 02 15:10:44.388789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:44.388739 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b"} err="failed to get container status \"da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b\": rpc error: code = NotFound desc = could not find container \"da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b\": container with ID starting with da58e0c8897c401caa6afc1adbed943eb09642c27f397093e7bc01643907177b not found: ID does not exist" Nov 02 15:10:44.533284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:44.533250 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-tnm4n"] Nov 02 15:10:44.579195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:44.579158 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d46d8f988-tnm4n"] Nov 02 15:10:45.881137 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:45.881104 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:46.388893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:46.388862 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:46.454740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:46.454706 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e421f0ac-f84e-463e-9c8a-990e37270e02" path="/var/lib/kubelet/pods/e421f0ac-f84e-463e-9c8a-990e37270e02/volumes" Nov 02 15:10:48.065290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:48.065258 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:48.221174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:48.221136 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:50.368840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:50.368801 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:10:50.510990 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:10:50.510953 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:11.009843 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:11:11.009796 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:11:20.176789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:20.176754 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:21.885241 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:21.885210 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:22.044217 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.044171 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:22.520350 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.520316 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz"] Nov 02 15:11:22.520751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.520736 5433 cpu_manager.go:414] "containerMap: removing container" podUID="e421f0ac-f84e-463e-9c8a-990e37270e02" containerName="manager" Nov 02 15:11:22.520797 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.520753 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e421f0ac-f84e-463e-9c8a-990e37270e02" containerName="manager" Nov 02 15:11:22.520851 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.520842 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e421f0ac-f84e-463e-9c8a-990e37270e02" containerName="manager" Nov 02 15:11:22.525211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.525191 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.583780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.583752 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e98976-openshift-default-dockercfg-z9ssr\"" Nov 02 15:11:22.597226 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.597196 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.597375 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.597234 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb87da-7c2e-4924-b606-3f97d446fc21-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.597375 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.597264 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.597449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.597402 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zpdk\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-kube-api-access-2zpdk\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.597501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.597450 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.597564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.597542 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.597628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.597610 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.597680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.597643 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.597789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.597772 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.616588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.616552 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:22.631066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.631039 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz"] Nov 02 15:11:22.698525 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698494 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.698708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698540 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.698708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698559 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb87da-7c2e-4924-b606-3f97d446fc21-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.698708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698578 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.698884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698718 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zpdk\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-kube-api-access-2zpdk\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.698884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698756 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.698884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698811 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.698884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698842 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.698884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698866 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.699168 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.698966 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.699232 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.699184 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.699345 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.699319 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.699414 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.699352 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.699470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.699437 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb87da-7c2e-4924-b606-3f97d446fc21-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.700918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.700893 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.701039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.700988 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.770795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.770714 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zpdk\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-kube-api-access-2zpdk\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.773806 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.773772 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.835439 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.835404 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:22.843894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.843865 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8"] Nov 02 15:11:22.848429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.848407 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.850567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.850548 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:22.898199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.898168 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fce776-openshift-default-dockercfg-5zhzc\"" Nov 02 15:11:22.900672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.900638 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.900839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.900701 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.900839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.900744 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.900839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.900798 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.901024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.900835 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.901024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.900866 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2696aeeb-a326-4622-aafd-c7665e1269cc-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.901024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.900920 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.901024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.900948 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.901388 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.901358 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7zx\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-kube-api-access-7j7zx\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:22.910844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:22.910818 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8"] Nov 02 15:11:23.002484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.002443 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7zx\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-kube-api-access-7j7zx\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.002484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.002488 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.002742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.002515 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.002742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.002541 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.002742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.002580 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.002742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.002617 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.002742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.002644 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2696aeeb-a326-4622-aafd-c7665e1269cc-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.002742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.002695 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.002742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.002719 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.003146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.003051 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.003146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.003062 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.003508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.003483 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.003508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.003491 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2696aeeb-a326-4622-aafd-c7665e1269cc-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.003656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.003544 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.005072 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.005055 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.005188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.005171 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.007220 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.007186 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:23.065404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.065337 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.066143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.066125 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7zx\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-kube-api-access-7j7zx\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.164754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.164727 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:23.192741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.192713 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:23.298231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.298201 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz"] Nov 02 15:11:23.301290 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:23.301258 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76eb87da_7c2e_4924_b606_3f97d446fc21.slice/crio-1d4422d2ba65b39f3eedabcab067edbd679093e2b254bb4324bb802868bf20c1 WatchSource:0}: Error finding container 1d4422d2ba65b39f3eedabcab067edbd679093e2b254bb4324bb802868bf20c1: Status 404 returned error can't find the container with id 1d4422d2ba65b39f3eedabcab067edbd679093e2b254bb4324bb802868bf20c1 Nov 02 15:11:23.303656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.303627 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:23.303736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.303689 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:23.303736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.303718 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:23.379779 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.379753 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:23.545776 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.543673 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8"] Nov 02 15:11:23.552219 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:23.552190 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2696aeeb_a326_4622_aafd_c7665e1269cc.slice/crio-9069500f06f62e48a8f2733ead06628edea37007982bc2d29de66b9e192817c6 WatchSource:0}: Error finding container 9069500f06f62e48a8f2733ead06628edea37007982bc2d29de66b9e192817c6: Status 404 returned error can't find the container with id 9069500f06f62e48a8f2733ead06628edea37007982bc2d29de66b9e192817c6 Nov 02 15:11:23.554856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.554815 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:23.554964 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.554902 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:23.554964 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.554950 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:23.558958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.558934 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" event={"ID":"76eb87da-7c2e-4924-b606-3f97d446fc21","Type":"ContainerStarted","Data":"e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9"} Nov 02 15:11:23.559043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.558996 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" event={"ID":"76eb87da-7c2e-4924-b606-3f97d446fc21","Type":"ContainerStarted","Data":"1d4422d2ba65b39f3eedabcab067edbd679093e2b254bb4324bb802868bf20c1"} Nov 02 15:11:23.567311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.567278 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw"] Nov 02 15:11:23.572534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.572513 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.577697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.577673 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:23.629189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.628931 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-71d1eb-openshift-default-dockercfg-zn6lk\"" Nov 02 15:11:23.643005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.639858 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw"] Nov 02 15:11:23.688422 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.688359 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" podStartSLOduration=1.688339793 podStartE2EDuration="1.688339793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:23.686219009 +0000 UTC m=+787.359422204" watchObservedRunningTime="2026-11-02 15:11:23.688339793 +0000 UTC m=+787.361542990" Nov 02 15:11:23.710198 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.710162 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599mf\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-kube-api-access-599mf\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.710367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.710205 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144b572d-c99c-4ea7-8b25-3a51bcd20421-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.710367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.710237 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.710367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.710327 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.710367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.710366 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.710563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.710398 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.710563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.710428 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.710563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.710495 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.710563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.710532 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.749802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.749771 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:23.811608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.811575 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.811963 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.811642 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.812122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.812025 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.812122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.812059 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.812307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.812149 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.812307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.812193 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.812307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.812294 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.812476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.812416 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.812544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.812520 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.813725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.813692 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599mf\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-kube-api-access-599mf\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.813852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.813741 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144b572d-c99c-4ea7-8b25-3a51bcd20421-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.813852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.813777 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.814160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.814136 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.814325 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.814306 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144b572d-c99c-4ea7-8b25-3a51bcd20421-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.814812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.814795 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.814944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.814926 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.850814 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.850756 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:23.891536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.891511 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.899559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.899516 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-599mf\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-kube-api-access-599mf\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.905339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.905320 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:23.927520 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:23.927489 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:24.109765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.109672 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:24.287490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.287465 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw"] Nov 02 15:11:24.289897 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:24.289873 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144b572d_c99c_4ea7_8b25_3a51bcd20421.slice/crio-fe87e317852bef08538ebf8f4bbe91d11d2ba8d45a2b3a6558b8e50e214e10ae WatchSource:0}: Error finding container fe87e317852bef08538ebf8f4bbe91d11d2ba8d45a2b3a6558b8e50e214e10ae: Status 404 returned error can't find the container with id fe87e317852bef08538ebf8f4bbe91d11d2ba8d45a2b3a6558b8e50e214e10ae Nov 02 15:11:24.292306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.292273 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.292404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.292338 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.292404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.292366 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.359922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.359841 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:24.560327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.560290 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:24.568599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.568567 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" event={"ID":"2696aeeb-a326-4622-aafd-c7665e1269cc","Type":"ContainerStarted","Data":"fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20"} Nov 02 15:11:24.568781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.568629 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" event={"ID":"2696aeeb-a326-4622-aafd-c7665e1269cc","Type":"ContainerStarted","Data":"9069500f06f62e48a8f2733ead06628edea37007982bc2d29de66b9e192817c6"} Nov 02 15:11:24.570122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.570090 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" event={"ID":"144b572d-c99c-4ea7-8b25-3a51bcd20421","Type":"ContainerStarted","Data":"8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec"} Nov 02 15:11:24.570245 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.570141 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" event={"ID":"144b572d-c99c-4ea7-8b25-3a51bcd20421","Type":"ContainerStarted","Data":"fe87e317852bef08538ebf8f4bbe91d11d2ba8d45a2b3a6558b8e50e214e10ae"} Nov 02 15:11:24.698825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.698772 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" podStartSLOduration=2.698758263 podStartE2EDuration="2.698758263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:24.698206883 +0000 UTC m=+788.371410089" watchObservedRunningTime="2026-11-02 15:11:24.698758263 +0000 UTC m=+788.371961457" Nov 02 15:11:24.703108 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.703079 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc"] Nov 02 15:11:24.706700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.706677 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.734888 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.734855 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:24.758177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.758150 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-624908-openshift-default-dockercfg-zvxm2\"" Nov 02 15:11:24.768923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.768891 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc"] Nov 02 15:11:24.812414 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.812338 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" podStartSLOduration=1.812312496 podStartE2EDuration="1.812312496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:24.811354903 +0000 UTC m=+788.484558097" watchObservedRunningTime="2026-11-02 15:11:24.812312496 +0000 UTC m=+788.485515696" Nov 02 15:11:24.825186 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.825151 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.825331 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.825195 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4df3520-364b-4282-a066-a83c7e2ff592-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.825331 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.825290 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.825402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.825343 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.825435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.825403 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.825470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.825430 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8xn\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-kube-api-access-sf8xn\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.825470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.825458 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4df3520-364b-4282-a066-a83c7e2ff592-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.825560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.825477 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.825656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.825636 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.856256 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.856217 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:24.881924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.881890 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:24.906129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.906091 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:24.926593 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.926559 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.926770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.926613 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.926770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.926740 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4df3520-364b-4282-a066-a83c7e2ff592-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.926895 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.926802 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.926895 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.926880 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.927021 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.926916 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.927021 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.926945 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8xn\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-kube-api-access-sf8xn\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.927021 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.927012 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4df3520-364b-4282-a066-a83c7e2ff592-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.927174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.927042 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.927174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.927088 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.927174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.927109 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.927322 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.927293 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.927588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.927531 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.927823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.927802 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4df3520-364b-4282-a066-a83c7e2ff592-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.929147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.929125 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.929381 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.929351 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4df3520-364b-4282-a066-a83c7e2ff592-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.987829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.987751 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:24.999348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:24.999277 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8xn\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-kube-api-access-sf8xn\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:25.025146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.025104 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:25.040142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.040105 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:25.165594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.165562 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:25.170555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.170531 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:25.402273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.402172 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc"] Nov 02 15:11:25.405001 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:25.404956 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4df3520_364b_4282_a066_a83c7e2ff592.slice/crio-53c0c13071dad66865c813314c681c0b7c034ffc3670c68218a0e0a4caab88cb WatchSource:0}: Error finding container 53c0c13071dad66865c813314c681c0b7c034ffc3670c68218a0e0a4caab88cb: Status 404 returned error can't find the container with id 53c0c13071dad66865c813314c681c0b7c034ffc3670c68218a0e0a4caab88cb Nov 02 15:11:25.407398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.407365 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:25.407480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.407432 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:25.407480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.407462 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:25.576610 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.576576 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" event={"ID":"d4df3520-364b-4282-a066-a83c7e2ff592","Type":"ContainerStarted","Data":"903ac58cc6aabd07075250b8963819a3cd59fa62db60cc4b9aa64b20696d74b4"} Nov 02 15:11:25.576804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.576628 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" event={"ID":"d4df3520-364b-4282-a066-a83c7e2ff592","Type":"ContainerStarted","Data":"53c0c13071dad66865c813314c681c0b7c034ffc3670c68218a0e0a4caab88cb"} Nov 02 15:11:25.577781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.577388 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:25.577781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.577442 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:25.578069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.578042 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:11:25.578237 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.578223 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:11:25.695493 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.695362 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" podStartSLOduration=1.695337271 podStartE2EDuration="1.695337271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:25.693144962 +0000 UTC m=+789.366348158" watchObservedRunningTime="2026-11-02 15:11:25.695337271 +0000 UTC m=+789.368540493" Nov 02 15:11:25.911103 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:25.911072 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:26.025676 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:26.025597 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:26.031448 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:26.031421 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:26.581662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:26.581625 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:26.581850 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:26.581676 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:26.582262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:26.582239 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:11:26.582474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:26.582459 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:11:26.931511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:26.931474 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:27.107462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:27.107417 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:27.254656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:27.254564 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:28.019471 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:28.019436 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:28.214401 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:28.214367 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:28.370919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:28.370838 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:28.734329 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:28.734292 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:28.956198 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:28.956166 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:29.112026 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:29.111931 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:30.639327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:30.639296 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:30.816446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:30.816411 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:30.999088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:30.999001 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:31.176859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.176826 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:31.326412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.326331 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:31.530766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.530737 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:31.540680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.540644 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk"] Nov 02 15:11:31.544230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.544211 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:31.646511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.646472 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-e98976\"" Nov 02 15:11:31.646511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.646485 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-e98976-dockercfg-nb58t\"" Nov 02 15:11:31.648034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.648013 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk"] Nov 02 15:11:31.649284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.649261 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r"] Nov 02 15:11:31.652929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.652912 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:31.706833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.706793 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373b0af7-e073-474c-b465-8c0b22b359af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk\" (UID: \"373b0af7-e073-474c-b465-8c0b22b359af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:31.707109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.706965 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-798z5\" (UniqueName: \"kubernetes.io/projected/373b0af7-e073-474c-b465-8c0b22b359af-kube-api-access-798z5\") pod \"payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk\" (UID: \"373b0af7-e073-474c-b465-8c0b22b359af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:31.717868 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.717829 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:31.719654 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.719624 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r"] Nov 02 15:11:31.807929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.807892 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72h5x\" (UniqueName: \"kubernetes.io/projected/500c39b4-df23-4fe6-abeb-46b7b9172de2-kube-api-access-72h5x\") pod \"payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r\" (UID: \"500c39b4-df23-4fe6-abeb-46b7b9172de2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:31.808109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.807948 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-798z5\" (UniqueName: \"kubernetes.io/projected/373b0af7-e073-474c-b465-8c0b22b359af-kube-api-access-798z5\") pod \"payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk\" (UID: \"373b0af7-e073-474c-b465-8c0b22b359af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:31.808157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.808113 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373b0af7-e073-474c-b465-8c0b22b359af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk\" (UID: \"373b0af7-e073-474c-b465-8c0b22b359af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:31.808157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.808148 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500c39b4-df23-4fe6-abeb-46b7b9172de2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r\" (UID: \"500c39b4-df23-4fe6-abeb-46b7b9172de2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:31.808768 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.808751 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373b0af7-e073-474c-b465-8c0b22b359af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk\" (UID: \"373b0af7-e073-474c-b465-8c0b22b359af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:31.865828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.865797 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-798z5\" (UniqueName: \"kubernetes.io/projected/373b0af7-e073-474c-b465-8c0b22b359af-kube-api-access-798z5\") pod \"payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk\" (UID: \"373b0af7-e073-474c-b465-8c0b22b359af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:31.909356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.909244 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500c39b4-df23-4fe6-abeb-46b7b9172de2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r\" (UID: \"500c39b4-df23-4fe6-abeb-46b7b9172de2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:31.909498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.909366 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72h5x\" (UniqueName: \"kubernetes.io/projected/500c39b4-df23-4fe6-abeb-46b7b9172de2-kube-api-access-72h5x\") pod \"payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r\" (UID: \"500c39b4-df23-4fe6-abeb-46b7b9172de2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:31.910129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.910099 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500c39b4-df23-4fe6-abeb-46b7b9172de2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r\" (UID: \"500c39b4-df23-4fe6-abeb-46b7b9172de2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:31.982357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:31.982326 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72h5x\" (UniqueName: \"kubernetes.io/projected/500c39b4-df23-4fe6-abeb-46b7b9172de2-kube-api-access-72h5x\") pod \"payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r\" (UID: \"500c39b4-df23-4fe6-abeb-46b7b9172de2\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:32.157793 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:32.157764 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:32.266677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:32.266646 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:32.522469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:32.522445 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk"] Nov 02 15:11:32.524839 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:32.524810 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373b0af7_e073_474c_b465_8c0b22b359af.slice/crio-b257b7e2dfbd1f5680102228b2557aec8f7e61e77fd7d180396f191615097b20 WatchSource:0}: Error finding container b257b7e2dfbd1f5680102228b2557aec8f7e61e77fd7d180396f191615097b20: Status 404 returned error can't find the container with id b257b7e2dfbd1f5680102228b2557aec8f7e61e77fd7d180396f191615097b20 Nov 02 15:11:32.616221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:32.616189 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" event={"ID":"373b0af7-e073-474c-b465-8c0b22b359af","Type":"ContainerStarted","Data":"4d793d05a38e8a1efa198326dda70bcecc3d3f47e35ea023f94c23e9f40353d6"} Nov 02 15:11:32.616400 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:32.616273 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" event={"ID":"373b0af7-e073-474c-b465-8c0b22b359af","Type":"ContainerStarted","Data":"b257b7e2dfbd1f5680102228b2557aec8f7e61e77fd7d180396f191615097b20"} Nov 02 15:11:32.616400 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:32.616343 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:32.629345 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:32.629319 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r"] Nov 02 15:11:32.632420 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:32.632398 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500c39b4_df23_4fe6_abeb_46b7b9172de2.slice/crio-f519aba9e9fa059079db9bb84fb80a01a03329700670e72dd2a70de96ef80637 WatchSource:0}: Error finding container f519aba9e9fa059079db9bb84fb80a01a03329700670e72dd2a70de96ef80637: Status 404 returned error can't find the container with id f519aba9e9fa059079db9bb84fb80a01a03329700670e72dd2a70de96ef80637 Nov 02 15:11:32.744449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:32.744385 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" podStartSLOduration=1.744368476 podStartE2EDuration="1.744368476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:32.743559403 +0000 UTC m=+796.416762599" watchObservedRunningTime="2026-11-02 15:11:32.744368476 +0000 UTC m=+796.417571672" Nov 02 15:11:33.622524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.622477 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" event={"ID":"500c39b4-df23-4fe6-abeb-46b7b9172de2","Type":"ContainerStarted","Data":"ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444"} Nov 02 15:11:33.622688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.622546 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" event={"ID":"500c39b4-df23-4fe6-abeb-46b7b9172de2","Type":"ContainerStarted","Data":"f519aba9e9fa059079db9bb84fb80a01a03329700670e72dd2a70de96ef80637"} Nov 02 15:11:33.622688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.622628 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:33.734902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.734835 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" podStartSLOduration=2.7348169589999998 podStartE2EDuration="2.734816959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:33.734379722 +0000 UTC m=+797.407582919" watchObservedRunningTime="2026-11-02 15:11:33.734816959 +0000 UTC m=+797.408020154" Nov 02 15:11:33.802300 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.802267 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8"] Nov 02 15:11:33.805828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.805804 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:33.861326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.861289 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8a2e6f-serving-cert\"" Nov 02 15:11:33.868748 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.868704 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8"] Nov 02 15:11:33.929406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.929371 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-metrics-certs\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:33.929581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.929437 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:33.929581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.929525 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2d51c-760f-447d-8a31-675d6dae2424-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:33.929581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:33.929554 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjdt6\" (UniqueName: \"kubernetes.io/projected/9dc2d51c-760f-447d-8a31-675d6dae2424-kube-api-access-wjdt6\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.011194 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.011162 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:34.031112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.031083 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-metrics-certs\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.031293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.031145 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.031293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.031201 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2d51c-760f-447d-8a31-675d6dae2424-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.031293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.031225 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjdt6\" (UniqueName: \"kubernetes.io/projected/9dc2d51c-760f-447d-8a31-675d6dae2424-kube-api-access-wjdt6\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.031293 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:11:34.031252 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-serving-cert: secret "maas-api-e2e-worker-w4-8a2e6f-serving-cert" not found Nov 02 15:11:34.031553 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:11:34.031328 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls podName:9dc2d51c-760f-447d-8a31-675d6dae2424 nodeName:}" failed. No retries permitted until 2026-11-02 15:11:34.531304061 +0000 UTC m=+798.204507252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls") pod "maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" (UID: "9dc2d51c-760f-447d-8a31-675d6dae2424") : secret "maas-api-e2e-worker-w4-8a2e6f-serving-cert" not found Nov 02 15:11:34.032008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.031963 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2d51c-760f-447d-8a31-675d6dae2424-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.033567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.033539 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-metrics-certs\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.095843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.095804 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjdt6\" (UniqueName: \"kubernetes.io/projected/9dc2d51c-760f-447d-8a31-675d6dae2424-kube-api-access-wjdt6\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.188289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.188195 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:34.452282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.452194 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:34.536100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.536071 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.538517 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.538499 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:34.719111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:34.719009 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:35.089499 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:35.089413 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8"] Nov 02 15:11:35.092491 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:35.092462 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc2d51c_760f_447d_8a31_675d6dae2424.slice/crio-75034a385d63b40bd1c28628397d84c68f476bf76a7b935224fa7bf074fbdf25 WatchSource:0}: Error finding container 75034a385d63b40bd1c28628397d84c68f476bf76a7b935224fa7bf074fbdf25: Status 404 returned error can't find the container with id 75034a385d63b40bd1c28628397d84c68f476bf76a7b935224fa7bf074fbdf25 Nov 02 15:11:35.632807 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:35.632771 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" event={"ID":"9dc2d51c-760f-447d-8a31-675d6dae2424","Type":"ContainerStarted","Data":"75034a385d63b40bd1c28628397d84c68f476bf76a7b935224fa7bf074fbdf25"} Nov 02 15:11:35.954567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:35.954476 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:36.168245 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:36.168218 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:36.639340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:36.639298 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" event={"ID":"9dc2d51c-760f-447d-8a31-675d6dae2424","Type":"ContainerStarted","Data":"b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd"} Nov 02 15:11:36.639545 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:36.639435 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:36.751241 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:36.751178 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" podStartSLOduration=2.758503513 podStartE2EDuration="3.751162008s" totalImagesPullingTime="992.658495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:36.74976011 +0000 UTC m=+800.422963306" watchObservedRunningTime="2026-11-02 15:11:36.751162008 +0000 UTC m=+800.424365203" Nov 02 15:11:38.434311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.434274 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:38.595825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.595781 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:38.746414 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.746330 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:38.848338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.848298 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn"] Nov 02 15:11:38.852231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.852208 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:38.908347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.908319 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-fce776\"" Nov 02 15:11:38.917195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.917164 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:38.956338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.956307 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn"] Nov 02 15:11:38.958740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.958719 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-fce776-dockercfg-ssxml\"" Nov 02 15:11:38.992889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.992459 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfncb\" (UniqueName: \"kubernetes.io/projected/5d3aa650-c638-49cf-bcec-6b6201c977ec-kube-api-access-hfncb\") pod \"payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn\" (UID: \"5d3aa650-c638-49cf-bcec-6b6201c977ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:38.992889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:38.992574 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3aa650-c638-49cf-bcec-6b6201c977ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn\" (UID: \"5d3aa650-c638-49cf-bcec-6b6201c977ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:39.094121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.094033 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3aa650-c638-49cf-bcec-6b6201c977ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn\" (UID: \"5d3aa650-c638-49cf-bcec-6b6201c977ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:39.094257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.094132 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfncb\" (UniqueName: \"kubernetes.io/projected/5d3aa650-c638-49cf-bcec-6b6201c977ec-kube-api-access-hfncb\") pod \"payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn\" (UID: \"5d3aa650-c638-49cf-bcec-6b6201c977ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:39.094731 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.094710 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3aa650-c638-49cf-bcec-6b6201c977ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn\" (UID: \"5d3aa650-c638-49cf-bcec-6b6201c977ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:39.156518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.156492 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfncb\" (UniqueName: \"kubernetes.io/projected/5d3aa650-c638-49cf-bcec-6b6201c977ec-kube-api-access-hfncb\") pod \"payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn\" (UID: \"5d3aa650-c638-49cf-bcec-6b6201c977ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:39.164889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.164863 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:39.538987 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.538803 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn"] Nov 02 15:11:39.542472 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:39.542438 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3aa650_c638_49cf_bcec_6b6201c977ec.slice/crio-ae2b5114062dbc058298b3be06e8b15dd568276630d8c627d9fba5c02561c53f WatchSource:0}: Error finding container ae2b5114062dbc058298b3be06e8b15dd568276630d8c627d9fba5c02561c53f: Status 404 returned error can't find the container with id ae2b5114062dbc058298b3be06e8b15dd568276630d8c627d9fba5c02561c53f Nov 02 15:11:39.587730 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.587703 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc"] Nov 02 15:11:39.592330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.592306 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.652834 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.652771 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc"] Nov 02 15:11:39.658253 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.658218 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" event={"ID":"5d3aa650-c638-49cf-bcec-6b6201c977ec","Type":"ContainerStarted","Data":"79da29b57a79c048f81e73703333afcd28a7c677041b029a529878969d5cd932"} Nov 02 15:11:39.658515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.658285 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:39.658515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.658311 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" event={"ID":"5d3aa650-c638-49cf-bcec-6b6201c977ec","Type":"ContainerStarted","Data":"ae2b5114062dbc058298b3be06e8b15dd568276630d8c627d9fba5c02561c53f"} Nov 02 15:11:39.700446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.700406 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424e9afc-422a-4b4d-a56d-af2a9973e329-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.700671 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.700643 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.700751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.700683 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-metrics-certs\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.700751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.700710 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjngb\" (UniqueName: \"kubernetes.io/projected/424e9afc-422a-4b4d-a56d-af2a9973e329-kube-api-access-rjngb\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.722454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.722424 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-fce776-serving-cert\"" Nov 02 15:11:39.802213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.802113 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.802213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.802159 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-metrics-certs\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.802213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.802179 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjngb\" (UniqueName: \"kubernetes.io/projected/424e9afc-422a-4b4d-a56d-af2a9973e329-kube-api-access-rjngb\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.802213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.802205 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424e9afc-422a-4b4d-a56d-af2a9973e329-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.802564 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:11:39.802241 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-serving-cert: secret "maas-api-e2e-worker-w2-fce776-serving-cert" not found Nov 02 15:11:39.802564 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:11:39.802366 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls podName:424e9afc-422a-4b4d-a56d-af2a9973e329 nodeName:}" failed. No retries permitted until 2026-11-02 15:11:40.302342859 +0000 UTC m=+803.975546047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls") pod "maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" (UID: "424e9afc-422a-4b4d-a56d-af2a9973e329") : secret "maas-api-e2e-worker-w2-fce776-serving-cert" not found Nov 02 15:11:39.802916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.802899 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424e9afc-422a-4b4d-a56d-af2a9973e329-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.804550 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.804529 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-metrics-certs\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:39.813506 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.813449 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" podStartSLOduration=1.813432641 podStartE2EDuration="1.813432641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:39.811414458 +0000 UTC m=+803.484617652" watchObservedRunningTime="2026-11-02 15:11:39.813432641 +0000 UTC m=+803.486635837" Nov 02 15:11:39.822852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.822821 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:39.863090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:39.863055 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjngb\" (UniqueName: \"kubernetes.io/projected/424e9afc-422a-4b4d-a56d-af2a9973e329-kube-api-access-rjngb\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:40.014549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:40.014511 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:40.168729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:40.168700 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:40.307336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:40.307287 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:40.309759 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:40.309732 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fce776-b7b69d65-624fc\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:40.511394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:40.511310 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:40.939366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:40.939337 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc"] Nov 02 15:11:40.941483 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:40.941453 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424e9afc_422a_4b4d_a56d_af2a9973e329.slice/crio-3dbffbe61e4523497c2781cf186ecbd807dbd7cb840d13b35fc36e6173c156ba WatchSource:0}: Error finding container 3dbffbe61e4523497c2781cf186ecbd807dbd7cb840d13b35fc36e6173c156ba: Status 404 returned error can't find the container with id 3dbffbe61e4523497c2781cf186ecbd807dbd7cb840d13b35fc36e6173c156ba Nov 02 15:11:40.962301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:40.962267 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:41.027648 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:11:41.027614 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:11:41.125296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:41.125260 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:41.279333 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:41.279246 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:41.668760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:41.668720 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" event={"ID":"424e9afc-422a-4b4d-a56d-af2a9973e329","Type":"ContainerStarted","Data":"3dbffbe61e4523497c2781cf186ecbd807dbd7cb840d13b35fc36e6173c156ba"} Nov 02 15:11:42.649808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:42.649781 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:11:42.674804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:42.674770 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" event={"ID":"424e9afc-422a-4b4d-a56d-af2a9973e329","Type":"ContainerStarted","Data":"86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc"} Nov 02 15:11:42.674959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:42.674827 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:42.873958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:42.873895 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" podStartSLOduration=3.031098004 podStartE2EDuration="3.873875169s" totalImagesPullingTime="842.777165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:42.873560097 +0000 UTC m=+806.546763293" watchObservedRunningTime="2026-11-02 15:11:42.873875169 +0000 UTC m=+806.547078366" Nov 02 15:11:43.439390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:43.439351 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:43.610741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:43.610708 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:43.626490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:43.626466 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:11:44.630141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:44.630111 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:11:45.292907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.292876 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b"] Nov 02 15:11:45.296635 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.296614 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.345188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.345152 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-71d1eb-serving-cert\"" Nov 02 15:11:45.357488 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.357440 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-metrics-certs\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.357667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.357496 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbq7\" (UniqueName: \"kubernetes.io/projected/d0638183-955f-4411-8f7b-49c24124e8a6-kube-api-access-zgbq7\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.357667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.357629 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.357776 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.357681 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0638183-955f-4411-8f7b-49c24124e8a6-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.393105 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.393073 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b"] Nov 02 15:11:45.458763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.458729 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-metrics-certs\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.458763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.458769 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbq7\" (UniqueName: \"kubernetes.io/projected/d0638183-955f-4411-8f7b-49c24124e8a6-kube-api-access-zgbq7\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.459062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.458791 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.459062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.458810 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0638183-955f-4411-8f7b-49c24124e8a6-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.459062 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:11:45.458902 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-serving-cert: secret "maas-api-e2e-worker-w3-71d1eb-serving-cert" not found Nov 02 15:11:45.459062 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:11:45.459007 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls podName:d0638183-955f-4411-8f7b-49c24124e8a6 nodeName:}" failed. No retries permitted until 2026-11-02 15:11:45.958956687 +0000 UTC m=+809.632159867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls") pod "maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" (UID: "d0638183-955f-4411-8f7b-49c24124e8a6") : secret "maas-api-e2e-worker-w3-71d1eb-serving-cert" not found Nov 02 15:11:45.459458 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.459441 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0638183-955f-4411-8f7b-49c24124e8a6-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.461206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.461184 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-metrics-certs\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.518556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.518522 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbq7\" (UniqueName: \"kubernetes.io/projected/d0638183-955f-4411-8f7b-49c24124e8a6-kube-api-access-zgbq7\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.557717 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.557624 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:45.781336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.781294 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:45.929989 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.929937 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:45.964469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.964439 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:45.966867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:45.966840 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:46.209666 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:46.209581 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:46.584909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:46.584884 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b"] Nov 02 15:11:46.587057 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:11:46.587029 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0638183_955f_4411_8f7b_49c24124e8a6.slice/crio-6ff44db5e6c7b5885a72d277324a9458f6e1aafaa9abb233c4b779d7f263a929 WatchSource:0}: Error finding container 6ff44db5e6c7b5885a72d277324a9458f6e1aafaa9abb233c4b779d7f263a929: Status 404 returned error can't find the container with id 6ff44db5e6c7b5885a72d277324a9458f6e1aafaa9abb233c4b779d7f263a929 Nov 02 15:11:46.695236 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:46.695192 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" event={"ID":"d0638183-955f-4411-8f7b-49c24124e8a6","Type":"ContainerStarted","Data":"6ff44db5e6c7b5885a72d277324a9458f6e1aafaa9abb233c4b779d7f263a929"} Nov 02 15:11:47.701082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:47.701044 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" event={"ID":"d0638183-955f-4411-8f7b-49c24124e8a6","Type":"ContainerStarted","Data":"2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9"} Nov 02 15:11:47.701486 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:47.701147 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:48.187127 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:48.187093 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:48.369570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:48.369536 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:48.688617 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:48.688377 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:11:48.806315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:48.806264 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" podStartSLOduration=2.839573051 podStartE2EDuration="3.806250469s" totalImagesPullingTime="966.677418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:47.838771045 +0000 UTC m=+811.511974241" watchObservedRunningTime="2026-11-02 15:11:48.806250469 +0000 UTC m=+812.479453727" Nov 02 15:11:50.627958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:50.627919 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:50.665730 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:50.665706 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:11:50.823703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:50.823667 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:51.349590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:51.349548 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:51.535675 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:51.535586 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:51.681252 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:51.681217 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:53.715540 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:53.715514 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:11:55.548819 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:55.548781 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:55.738999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:55.736565 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:11:56.838002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:11:56.837905 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:00.793833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:00.793798 5433 generic.go:356] "Generic (PLEG): container finished" podID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerID="8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec" exitCode=0 Nov 02 15:12:00.794314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:00.793869 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" event={"ID":"144b572d-c99c-4ea7-8b25-3a51bcd20421","Type":"ContainerDied","Data":"8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec"} Nov 02 15:12:00.794314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:00.794283 5433 scope.go:122] "RemoveContainer" containerID="8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec" Nov 02 15:12:00.795590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:00.795564 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:00.795665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:00.795633 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:00.795705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:00.795665 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:01.800788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.800754 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" event={"ID":"144b572d-c99c-4ea7-8b25-3a51bcd20421","Type":"ContainerStarted","Data":"abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188"} Nov 02 15:12:01.903466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.903429 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz"] Nov 02 15:12:01.906524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.906495 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:01.906654 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.906544 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 15:12:01.907964 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.907949 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:12:01.908067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.908052 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:01.934996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.920149 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz"] Nov 02 15:12:01.976881 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.976596 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-891723-serving-cert\"" Nov 02 15:12:01.976881 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.976736 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-metrics-certs\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:01.976881 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.976814 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:01.977273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.976930 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbcmc\" (UniqueName: \"kubernetes.io/projected/1193425a-4b6b-47ac-827f-20a00f789df1-kube-api-access-gbcmc\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:01.977273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:01.976965 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1193425a-4b6b-47ac-827f-20a00f789df1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.074753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.074679 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:02.077930 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.077892 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbcmc\" (UniqueName: \"kubernetes.io/projected/1193425a-4b6b-47ac-827f-20a00f789df1-kube-api-access-gbcmc\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.077930 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.077928 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1193425a-4b6b-47ac-827f-20a00f789df1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.078181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.078002 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-metrics-certs\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.078181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.078040 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.078181 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:12:02.078144 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-serving-cert: secret "maas-api-e2e-worker-w5-891723-serving-cert" not found Nov 02 15:12:02.078339 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:12:02.078189 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls podName:1193425a-4b6b-47ac-827f-20a00f789df1 nodeName:}" failed. No retries permitted until 2026-11-02 15:12:02.57817502 +0000 UTC m=+826.251378197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls") pod "maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" (UID: "1193425a-4b6b-47ac-827f-20a00f789df1") : secret "maas-api-e2e-worker-w5-891723-serving-cert" not found Nov 02 15:12:02.078706 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.078683 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1193425a-4b6b-47ac-827f-20a00f789df1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.080949 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.080928 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-metrics-certs\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.177938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.177902 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbcmc\" (UniqueName: \"kubernetes.io/projected/1193425a-4b6b-47ac-827f-20a00f789df1-kube-api-access-gbcmc\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.248468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.248430 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:02.395116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.395086 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:02.582960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.582921 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.587025 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.585393 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.837053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.833180 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:02.906641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.906608 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:02.906831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:02.906657 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 15:12:03.278373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:03.278345 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz"] Nov 02 15:12:03.280061 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:12:03.280031 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1193425a_4b6b_47ac_827f_20a00f789df1.slice/crio-7385844c846b8193ed0d1e6a54f71f7631060b43c79c2dcdd701bbed4c95f989 WatchSource:0}: Error finding container 7385844c846b8193ed0d1e6a54f71f7631060b43c79c2dcdd701bbed4c95f989: Status 404 returned error can't find the container with id 7385844c846b8193ed0d1e6a54f71f7631060b43c79c2dcdd701bbed4c95f989 Nov 02 15:12:03.867332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:03.867278 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" event={"ID":"1193425a-4b6b-47ac-827f-20a00f789df1","Type":"ContainerStarted","Data":"7385844c846b8193ed0d1e6a54f71f7631060b43c79c2dcdd701bbed4c95f989"} Nov 02 15:12:03.906489 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:03.906452 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:03.906653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:03.906513 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 15:12:04.875958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:04.875899 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" event={"ID":"1193425a-4b6b-47ac-827f-20a00f789df1","Type":"ContainerStarted","Data":"406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1"} Nov 02 15:12:04.876358 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:04.876071 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:04.906278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:04.906248 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:04.906426 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:04.906297 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 15:12:04.997781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:04.997706 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" podStartSLOduration=3.08185633 podStartE2EDuration="3.997679854s" totalImagesPullingTime="915.823524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:04.9958946 +0000 UTC m=+828.669097795" watchObservedRunningTime="2026-11-02 15:12:04.997679854 +0000 UTC m=+828.670883049" Nov 02 15:12:05.906306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:05.906267 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:05.906693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:05.906317 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 15:12:06.906132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:06.906094 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:06.906320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:06.906163 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 15:12:07.285739 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.285657 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6"] Nov 02 15:12:07.289431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.289409 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.338039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.338007 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-61a835-serving-cert\"" Nov 02 15:12:07.351825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.351783 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.352024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.351895 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj8mp\" (UniqueName: \"kubernetes.io/projected/93c9a70c-cb34-41bc-a936-0a9950627165-kube-api-access-wj8mp\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.352024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.352018 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c9a70c-cb34-41bc-a936-0a9950627165-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.352152 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.352059 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-metrics-certs\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.385900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.385860 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6"] Nov 02 15:12:07.453635 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.453597 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c9a70c-cb34-41bc-a936-0a9950627165-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.453868 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.453658 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-metrics-certs\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.453868 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.453730 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.453868 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.453823 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8mp\" (UniqueName: \"kubernetes.io/projected/93c9a70c-cb34-41bc-a936-0a9950627165-kube-api-access-wj8mp\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.454044 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:12:07.453918 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-serving-cert: secret "maas-api-e2e-worker-w0-61a835-serving-cert" not found Nov 02 15:12:07.454044 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:12:07.454027 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls podName:93c9a70c-cb34-41bc-a936-0a9950627165 nodeName:}" failed. No retries permitted until 2026-11-02 15:12:07.95397048 +0000 UTC m=+831.627173676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls") pod "maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" (UID: "93c9a70c-cb34-41bc-a936-0a9950627165") : secret "maas-api-e2e-worker-w0-61a835-serving-cert" not found Nov 02 15:12:07.454552 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.454528 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c9a70c-cb34-41bc-a936-0a9950627165-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.456847 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.456821 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-metrics-certs\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.496350 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.496312 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:07.513583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.513554 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj8mp\" (UniqueName: \"kubernetes.io/projected/93c9a70c-cb34-41bc-a936-0a9950627165-kube-api-access-wj8mp\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.672441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.672411 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:07.884922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.884884 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:07.910435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.910407 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:12:07.910807 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.910788 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:12:07.911464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.911444 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:12:07.958268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.958191 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:07.960843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:07.960809 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls\") pod \"maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:08.203474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:08.203437 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:08.575820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:08.575789 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6"] Nov 02 15:12:08.577573 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:12:08.577539 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c9a70c_cb34_41bc_a936_0a9950627165.slice/crio-d593a0f054f979deba7ebae36105affb3a8da0c562022bca7726ed905176fb01 WatchSource:0}: Error finding container d593a0f054f979deba7ebae36105affb3a8da0c562022bca7726ed905176fb01: Status 404 returned error can't find the container with id d593a0f054f979deba7ebae36105affb3a8da0c562022bca7726ed905176fb01 Nov 02 15:12:08.898672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:08.898632 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" event={"ID":"93c9a70c-cb34-41bc-a936-0a9950627165","Type":"ContainerStarted","Data":"d593a0f054f979deba7ebae36105affb3a8da0c562022bca7726ed905176fb01"} Nov 02 15:12:09.905291 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:09.905256 5433 generic.go:356] "Generic (PLEG): container finished" podID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerID="e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9" exitCode=0 Nov 02 15:12:09.905847 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:09.905332 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" event={"ID":"76eb87da-7c2e-4924-b606-3f97d446fc21","Type":"ContainerDied","Data":"e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9"} Nov 02 15:12:09.905847 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:09.905822 5433 scope.go:122] "RemoveContainer" containerID="e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9" Nov 02 15:12:09.907097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:09.907060 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" event={"ID":"93c9a70c-cb34-41bc-a936-0a9950627165","Type":"ContainerStarted","Data":"f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50"} Nov 02 15:12:09.907318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:09.907290 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:09.907399 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:09.907382 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:09.907451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:09.907437 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:10.577562 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:10.577523 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:12:10.887233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:10.887197 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:12:10.915128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:10.915094 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" event={"ID":"76eb87da-7c2e-4924-b606-3f97d446fc21","Type":"ContainerStarted","Data":"de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1"} Nov 02 15:12:10.915527 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:10.915289 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:11.004318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:11.004257 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" podStartSLOduration=3.133161562 podStartE2EDuration="4.004241597s" totalImagesPullingTime="871.080035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:10.175461563 +0000 UTC m=+833.848664782" watchObservedRunningTime="2026-11-02 15:12:11.004241597 +0000 UTC m=+834.677444793" Nov 02 15:12:11.054547 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:12:11.054506 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:12:11.850781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:11.850740 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:12:11.852036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:11.852010 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:12:11.852161 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:11.852060 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 15:12:12.850913 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:12.850871 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:12:12.851434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:12.850930 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 15:12:12.953542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:12.953503 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:13.851928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:13.851890 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:12:13.852352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:13.851946 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 15:12:14.851160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:14.851126 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:12:14.851343 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:14.851178 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 15:12:15.851118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:15.851083 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:12:15.851621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:15.851145 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 15:12:16.856534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:16.856497 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:12:16.857049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:16.856796 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:12:16.857573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:16.857556 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:12:16.951493 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:16.951466 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:12:18.014237 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:18.014189 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:18.183612 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:18.183572 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:18.337466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:18.337359 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:20.510034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:20.509941 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:23.413422 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:23.413380 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:23.689174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:23.689074 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:24.041590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:24.041475 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:24.204440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:24.204371 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:24.362359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:24.362269 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:28.579582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:28.579543 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:29.484294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:29.484257 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:33.859565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:33.859510 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:34.460855 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:34.460817 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:34.670079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:34.669964 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:34.840397 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:34.840354 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:35.278161 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:35.278121 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:36.972022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:36.971986 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:37.572667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:37.572607 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:37.745550 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:37.745518 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:37.919853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:37.919818 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:38.117278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.117244 5433 generic.go:356] "Generic (PLEG): container finished" podID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerID="fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20" exitCode=0 Nov 02 15:12:38.117698 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.117317 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" event={"ID":"2696aeeb-a326-4622-aafd-c7665e1269cc","Type":"ContainerDied","Data":"fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20"} Nov 02 15:12:38.117812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.117788 5433 scope.go:122] "RemoveContainer" containerID="fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20" Nov 02 15:12:38.119090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.119067 5433 generic.go:356] "Generic (PLEG): container finished" podID="d4df3520-364b-4282-a066-a83c7e2ff592" containerID="903ac58cc6aabd07075250b8963819a3cd59fa62db60cc4b9aa64b20696d74b4" exitCode=0 Nov 02 15:12:38.119189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.119111 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" event={"ID":"d4df3520-364b-4282-a066-a83c7e2ff592","Type":"ContainerDied","Data":"903ac58cc6aabd07075250b8963819a3cd59fa62db60cc4b9aa64b20696d74b4"} Nov 02 15:12:38.119258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.119236 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:38.119313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.119299 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:38.119368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.119336 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:38.119518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.119501 5433 scope.go:122] "RemoveContainer" containerID="903ac58cc6aabd07075250b8963819a3cd59fa62db60cc4b9aa64b20696d74b4" Nov 02 15:12:38.120775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.120749 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:38.120873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.120821 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:38.120939 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:38.120869 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:39.169998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:39.167329 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" event={"ID":"d4df3520-364b-4282-a066-a83c7e2ff592","Type":"ContainerStarted","Data":"99958702c4af86ad6a260f9483094fdc0358338baf8429814b112f218bb879b8"} Nov 02 15:12:39.173581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:39.173555 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" event={"ID":"2696aeeb-a326-4622-aafd-c7665e1269cc","Type":"ContainerStarted","Data":"dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f"} Nov 02 15:12:40.031001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:40.027552 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:12:40.031001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:40.027678 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:40.031001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:40.027724 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 15:12:40.168602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:40.168569 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:12:40.169043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:40.169007 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:12:40.169168 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:40.169105 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 15:12:40.578136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:40.578096 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:12:41.010562 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:12:41.010524 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:12:41.039129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:41.039083 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:41.039434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:41.039400 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 15:12:41.167773 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:41.167338 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:12:41.167773 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:41.167407 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 15:12:41.272602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:41.272509 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:41.582359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:41.582262 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:12:42.033902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:42.033858 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:42.034128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:42.033933 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 15:12:42.166526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:42.166492 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:12:42.166791 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:42.166753 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 15:12:43.032000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:43.027005 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:43.032000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:43.027069 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 15:12:43.150580 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:43.150545 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:43.167570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:43.167534 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:12:43.167887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:43.167839 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 15:12:43.790692 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:43.790659 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:43.944017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:43.943682 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:44.037999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:44.033506 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:44.037999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:44.033566 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 15:12:44.098051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:44.097914 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:44.165926 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:44.165889 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:12:44.166097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:44.165961 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 15:12:45.030053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:45.030019 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:12:45.030924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:45.030897 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:12:45.169461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:45.169431 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:12:45.170437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:45.170415 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:12:46.997201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:46.997164 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:48.825517 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:48.825484 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:52.474707 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:52.474670 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:55.123319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:55.123278 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:55.688923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:55.688889 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:56.447919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:56.447882 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:56.590631 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:56.590595 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:56.737783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:56.737702 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:58.153662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:58.153621 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:58.833406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:58.833371 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:59.437821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:59.437780 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:59.587640 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:59.587610 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:12:59.731650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:12:59.731565 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:02.327227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:02.327187 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:03.562394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:03.562349 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:08.761829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:08.761795 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:08.909556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:08.909502 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:11.011674 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:13:11.011638 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:13:11.032234 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:11.032205 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:11.665429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:11.665388 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:11.818700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:11.818662 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:11.959633 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:11.959545 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:14.077875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:14.077837 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:19.065887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:19.065849 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:20.847250 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:20.847217 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:24.058989 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:24.058954 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:26.407512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:26.407428 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:29.172788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:29.172753 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:33.933887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:33.933849 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:34.293751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:34.293672 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:39.033085 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:39.033045 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:39.354321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:39.354244 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:41.011025 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:13:41.010960 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:13:44.485449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:44.485418 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:48.313342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:48.313306 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:49.570068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:49.570032 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:54.655514 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:54.655480 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:56.839005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:56.838954 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:56.999875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:56.999841 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:13:59.695924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:13:59.695892 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:02.853717 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:02.853679 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:04.730933 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:04.730903 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:07.188516 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:07.188486 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:09.848099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:09.848062 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:11.011567 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:14:11.011530 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:14:14.959402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:14.959362 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:20.077927 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:20.077869 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:21.176382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:21.176348 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:25.065699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:25.065661 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:25.211839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:25.211800 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:29.154561 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:29.154524 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:29.310340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:29.310303 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:30.175637 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:30.175605 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:35.280549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:35.280505 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:36.190520 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.190486 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7"] Nov 02 15:14:36.193632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.193610 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.288138 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.288084 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"openshift-service-ca.crt\"" Nov 02 15:14:36.288596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.288174 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"default-dockercfg-xk8r7\"" Nov 02 15:14:36.288596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.288207 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"kube-root-ca.crt\"" Nov 02 15:14:36.288596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.288250 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"premium-w6-e98976-kserve-self-signed-certs\"" Nov 02 15:14:36.290305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.290280 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7"] Nov 02 15:14:36.291966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.291943 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:36.304341 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.304320 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tls-certs\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.304443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.304353 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh7t5\" (UniqueName: \"kubernetes.io/projected/9ee74c4b-83fe-42f7-980f-7fc179e9c569-kube-api-access-wh7t5\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.304443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.304379 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-dshm\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.304443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.304401 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-model-cache\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.304606 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.304584 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tmp-dir\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.304700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.304683 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-home\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.405404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.405371 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tmp-dir\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.405563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.405435 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-home\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.405563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.405512 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tls-certs\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.405563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.405547 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7t5\" (UniqueName: \"kubernetes.io/projected/9ee74c4b-83fe-42f7-980f-7fc179e9c569-kube-api-access-wh7t5\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.405684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.405577 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-dshm\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.405684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.405602 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-model-cache\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.405868 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.405849 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tmp-dir\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.405910 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.405898 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-home\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.405944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.405911 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-model-cache\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.407781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.407757 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-dshm\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.408202 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.408182 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tls-certs\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.463313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.463245 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh7t5\" (UniqueName: \"kubernetes.io/projected/9ee74c4b-83fe-42f7-980f-7fc179e9c569-kube-api-access-wh7t5\") pod \"premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.505132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.505108 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:36.876814 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:36.876788 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7"] Nov 02 15:14:36.879642 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:14:36.879610 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ee74c4b_83fe_42f7_980f_7fc179e9c569.slice/crio-e41e58f5a9504b485af9d732e6716bea4605c618b381b08dee6bf175a32edb51 WatchSource:0}: Error finding container e41e58f5a9504b485af9d732e6716bea4605c618b381b08dee6bf175a32edb51: Status 404 returned error can't find the container with id e41e58f5a9504b485af9d732e6716bea4605c618b381b08dee6bf175a32edb51 Nov 02 15:14:37.783188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:37.783147 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" event={"ID":"9ee74c4b-83fe-42f7-980f-7fc179e9c569","Type":"ContainerStarted","Data":"82562aff686ef6fdeeeedb44c0c6702a68b33fcd98c21906e729670348ce7510"} Nov 02 15:14:37.783644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:37.783200 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" event={"ID":"9ee74c4b-83fe-42f7-980f-7fc179e9c569","Type":"ContainerStarted","Data":"e41e58f5a9504b485af9d732e6716bea4605c618b381b08dee6bf175a32edb51"} Nov 02 15:14:37.893743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:37.893684 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podStartSLOduration=1.893670584 podStartE2EDuration="1.893670584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:37.893014424 +0000 UTC m=+981.566217618" watchObservedRunningTime="2026-11-02 15:14:37.893670584 +0000 UTC m=+981.566873778" Nov 02 15:14:38.505826 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:38.505798 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:40.362355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:40.362325 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:41.011196 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:14:41.011151 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:14:45.374833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:45.374799 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:46.468630 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.468601 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x"] Nov 02 15:14:46.471559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.471542 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.505917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.505884 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:46.518087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.518054 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x"] Nov 02 15:14:46.519563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.519526 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:46.561239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.561162 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"openshift-service-ca.crt\"" Nov 02 15:14:46.561239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.561185 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"kube-root-ca.crt\"" Nov 02 15:14:46.561537 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.561511 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:46.566501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.566481 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"default-dockercfg-sw5mb\"" Nov 02 15:14:46.566501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.566487 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"premium-w5-891723-kserve-self-signed-certs\"" Nov 02 15:14:46.619210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.619171 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-home\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.619586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.619559 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-dshm\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.623765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.619799 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tls-certs\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.623765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.622715 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-model-cache\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.623765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.622777 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-kube-api-access-pkg5x\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.623765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.622922 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tmp-dir\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.724060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.723956 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tmp-dir\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.724060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.724039 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-home\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.724277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.724066 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-dshm\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.724277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.724127 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tls-certs\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.724277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.724147 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-model-cache\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.724277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.724184 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-kube-api-access-pkg5x\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.724476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.724426 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tmp-dir\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.724476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.724453 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-home\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.724586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.724572 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-model-cache\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.726326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.726302 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-dshm\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.726824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.726798 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tls-certs\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.781682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.781644 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-kube-api-access-pkg5x\") pod \"premium-w5-891723-kserve-7fc4dcb469-jnr2x\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.783543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.783520 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:46.882878 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:46.882849 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:14:47.179083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:47.179052 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x"] Nov 02 15:14:47.181611 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:14:47.181574 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f5a7ca4_ac8d_4597_bf7d_9693df3bba1d.slice/crio-4c8c9fca1b9f7d4be814863a614ab485dca7e1e5d4297e27960542f4ad656196 WatchSource:0}: Error finding container 4c8c9fca1b9f7d4be814863a614ab485dca7e1e5d4297e27960542f4ad656196: Status 404 returned error can't find the container with id 4c8c9fca1b9f7d4be814863a614ab485dca7e1e5d4297e27960542f4ad656196 Nov 02 15:14:47.849740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:47.849698 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" event={"ID":"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d","Type":"ContainerStarted","Data":"0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2"} Nov 02 15:14:47.850154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:47.849750 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" event={"ID":"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d","Type":"ContainerStarted","Data":"4c8c9fca1b9f7d4be814863a614ab485dca7e1e5d4297e27960542f4ad656196"} Nov 02 15:14:47.964971 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:47.964919 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podStartSLOduration=1.964905865 podStartE2EDuration="1.964905865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:47.964704612 +0000 UTC m=+991.637907810" watchObservedRunningTime="2026-11-02 15:14:47.964905865 +0000 UTC m=+991.638109061" Nov 02 15:14:48.784489 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:48.784454 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:50.543622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:50.543577 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:55.596134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:55.596096 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:14:56.784501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:56.784422 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:56.797369 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:56.797338 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:14:56.908707 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:14:56.908676 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:15:00.797314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:00.797278 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:02.992149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:02.992116 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:05.867994 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:05.867950 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:10.756022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.755969 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc"] Nov 02 15:15:10.759087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.759065 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.768713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.768688 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.768862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.768725 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv57p\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-kube-api-access-vv57p\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.768862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.768753 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.768862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.768810 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.769041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.768878 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.769041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.769033 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.769124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.769070 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.769177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.769161 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.769246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.769225 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.808305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.808274 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-891723-openshift-default-dockercfg-c9wsk\"" Nov 02 15:15:10.856441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.856405 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc"] Nov 02 15:15:10.870354 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.870321 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.870548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.870375 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv57p\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-kube-api-access-vv57p\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.870548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.870419 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.870548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.870448 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.870548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.870501 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.870877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.870550 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.870877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.870583 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.870877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.870620 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.870877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.870649 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.871298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.871197 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.872014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.871461 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.872014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.871624 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.872196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.872020 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.872196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.872178 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.873641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.873618 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.873727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.873684 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.928383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.928348 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.928558 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.928495 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv57p\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-kube-api-access-vv57p\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:10.975070 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:10.975037 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:11.012244 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:15:11.012152 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:15:11.072615 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:11.072580 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:11.494326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:11.494302 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc"] Nov 02 15:15:11.497234 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:15:11.497205 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5115eacb_6b82_4a8e_bc75_b60b9c0fd83c.slice/crio-b50a3043dfd9e9df9e147336e4ae0c9efd001fbc3c3f34262f5df28f49047d5f WatchSource:0}: Error finding container b50a3043dfd9e9df9e147336e4ae0c9efd001fbc3c3f34262f5df28f49047d5f: Status 404 returned error can't find the container with id b50a3043dfd9e9df9e147336e4ae0c9efd001fbc3c3f34262f5df28f49047d5f Nov 02 15:15:11.499479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:11.499446 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:11.499567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:11.499510 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:11.499567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:11.499538 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:11.980853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:11.980808 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" event={"ID":"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c","Type":"ContainerStarted","Data":"1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44"} Nov 02 15:15:11.981402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:11.980911 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" event={"ID":"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c","Type":"ContainerStarted","Data":"b50a3043dfd9e9df9e147336e4ae0c9efd001fbc3c3f34262f5df28f49047d5f"} Nov 02 15:15:12.073600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:12.073561 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:12.075399 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:12.075370 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:15:12.075531 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:12.075424 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 15:15:13.073078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:13.073038 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:15:13.073556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:13.073098 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 15:15:14.072948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:14.072913 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:15:14.073153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:14.072996 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 15:15:15.073568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:15.073529 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:15:15.074081 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:15.073603 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 15:15:16.073248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:16.073205 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:15:16.073477 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:16.073278 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 15:15:16.179439 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:16.179402 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:17.073535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:17.073498 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 15:15:17.073723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:17.073562 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 15:15:18.211159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:18.211127 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:18.211626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:18.211445 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:18.212408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:18.212390 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:15:18.372604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:18.372555 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" podStartSLOduration=8.372540035 podStartE2EDuration="8.372540035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:12.18752824 +0000 UTC m=+1015.860731434" watchObservedRunningTime="2026-11-02 15:15:18.372540035 +0000 UTC m=+1022.045743230" Nov 02 15:15:21.273840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:21.273810 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:21.497761 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:21.497726 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:24.961942 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:24.961910 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:26.378358 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:26.378325 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:31.567430 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:31.567393 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:36.751880 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:36.751845 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:41.012916 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:15:41.012871 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:15:41.783224 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:41.783190 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:46.873677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:46.873643 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:52.058665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:52.058624 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:55.318212 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.318176 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv"] Nov 02 15:15:55.322007 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.321985 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.370748 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.370717 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8a2e6f-openshift-default-dockercfg-sqgd8\"" Nov 02 15:15:55.399076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.399037 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.399266 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.399083 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.399266 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.399101 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.399403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.399278 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.399403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.399347 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk7xz\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-kube-api-access-nk7xz\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.399480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.399440 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.399480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.399474 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.399627 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.399610 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.399718 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.399696 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.419302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.419267 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv"] Nov 02 15:15:55.501277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.501238 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.501459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.501291 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk7xz\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-kube-api-access-nk7xz\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.501459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.501320 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.501459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.501334 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.501459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.501380 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.501459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.501411 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.501706 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.501485 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.501706 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.501520 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.501706 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.501543 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.502418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.502304 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.502418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.502313 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.502418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.502369 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.502964 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.502701 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.503248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.503212 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.510745 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.505432 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.510745 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.505478 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.559520 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.559485 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.559713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.559689 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk7xz\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-kube-api-access-nk7xz\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:55.636018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:55.635968 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:56.052661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:56.052634 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv"] Nov 02 15:15:56.054690 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:15:56.054661 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode411e1cf_c1c8_4d31_84fd_91a4d0167b86.slice/crio-288bb543a2f29ae92973e6631eeb8fa5a1c28f7fe307435d948062c755fbe55e WatchSource:0}: Error finding container 288bb543a2f29ae92973e6631eeb8fa5a1c28f7fe307435d948062c755fbe55e: Status 404 returned error can't find the container with id 288bb543a2f29ae92973e6631eeb8fa5a1c28f7fe307435d948062c755fbe55e Nov 02 15:15:56.057129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:56.057092 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:56.057243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:56.057170 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:56.057243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:56.057217 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:56.226652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:56.226615 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" event={"ID":"e411e1cf-c1c8-4d31-84fd-91a4d0167b86","Type":"ContainerStarted","Data":"0782cea2eb169be3541c07639f7c10cdd0c2502f349fde5a5e405f053a079fdd"} Nov 02 15:15:56.226843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:56.226673 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" event={"ID":"e411e1cf-c1c8-4d31-84fd-91a4d0167b86","Type":"ContainerStarted","Data":"288bb543a2f29ae92973e6631eeb8fa5a1c28f7fe307435d948062c755fbe55e"} Nov 02 15:15:56.343150 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:56.343040 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" podStartSLOduration=1.343018905 podStartE2EDuration="1.343018905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:56.339564785 +0000 UTC m=+1060.012767979" watchObservedRunningTime="2026-11-02 15:15:56.343018905 +0000 UTC m=+1060.016222100" Nov 02 15:15:56.639146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:56.639066 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:15:57.208596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:57.208553 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:15:57.642786 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:57.642756 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:15:57.643177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:57.642811 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 15:15:58.637999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:58.636397 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:15:58.637999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:58.636462 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 15:15:59.636452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:59.636404 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:15:59.637164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:15:59.637089 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 15:16:00.637400 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:00.637363 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:16:00.638034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:00.638003 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 15:16:01.636395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:01.636357 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 02 15:16:01.636669 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:01.636641 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 02 15:16:02.415514 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:02.415475 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:02.780075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:02.779963 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:16:02.781326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:02.781299 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:16:02.782591 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:02.782569 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:16:07.479344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:07.479305 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:11.010619 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:16:11.010577 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:16:12.571700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:12.571667 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:17.708138 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:17.708106 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:21.826284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:21.826246 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:22.819855 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:22.819815 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:27.572856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:27.572768 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:27.883258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:27.883218 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:33.082332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:33.082284 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:34.832006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:34.831953 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:34.974143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:34.974106 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:36.372045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.372009 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2"] Nov 02 15:16:36.375787 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.375768 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.398461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.398431 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.398604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.398463 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.398604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.398487 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsngf\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-kube-api-access-lsngf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.398679 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.398656 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.398716 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.398692 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9da2d9-84e4-4805-863f-3b848377703d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.398814 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.398799 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.398857 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.398846 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.398907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.398895 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9da2d9-84e4-4805-863f-3b848377703d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.398942 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.398922 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.478411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.478374 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2"] Nov 02 15:16:36.499492 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499457 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.499659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499511 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.499659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499558 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9da2d9-84e4-4805-863f-3b848377703d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.499659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499594 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.499659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499645 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.499914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499749 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.499914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499780 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsngf\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-kube-api-access-lsngf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.499914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499860 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.499914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499889 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.499914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.499895 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9da2d9-84e4-4805-863f-3b848377703d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.500391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.500364 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9da2d9-84e4-4805-863f-3b848377703d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.500613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.500560 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.500613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.500580 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.500823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.500802 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.503230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.503206 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.503370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.503348 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9da2d9-84e4-4805-863f-3b848377703d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.557484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.557457 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.557739 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.557715 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsngf\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-kube-api-access-lsngf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-f5wz2\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:36.688702 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:36.688674 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:37.105166 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:37.104994 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2"] Nov 02 15:16:37.108579 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:16:37.108545 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9da2d9_84e4_4805_863f_3b848377703d.slice/crio-eb4a41e510a5afaf0518a028afe0d33cef95572d9c7498d4602a7f46ba7bb7e0 WatchSource:0}: Error finding container eb4a41e510a5afaf0518a028afe0d33cef95572d9c7498d4602a7f46ba7bb7e0: Status 404 returned error can't find the container with id eb4a41e510a5afaf0518a028afe0d33cef95572d9c7498d4602a7f46ba7bb7e0 Nov 02 15:16:37.110876 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:37.110844 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:37.110947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:37.110910 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:37.110947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:37.110942 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:37.499049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:37.499010 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" event={"ID":"da9da2d9-84e4-4805-863f-3b848377703d","Type":"ContainerStarted","Data":"22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc"} Nov 02 15:16:37.499472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:37.499077 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" event={"ID":"da9da2d9-84e4-4805-863f-3b848377703d","Type":"ContainerStarted","Data":"eb4a41e510a5afaf0518a028afe0d33cef95572d9c7498d4602a7f46ba7bb7e0"} Nov 02 15:16:37.615196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:37.615140 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" podStartSLOduration=1.6151267169999999 podStartE2EDuration="1.615126717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:37.612808974 +0000 UTC m=+1101.286012168" watchObservedRunningTime="2026-11-02 15:16:37.615126717 +0000 UTC m=+1101.288329912" Nov 02 15:16:37.688797 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:37.688763 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:38.197368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:38.197315 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:38.690725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:38.690687 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f5wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:16:38.691131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:38.690764 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 15:16:39.689585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:39.689554 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f5wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:16:39.689752 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:39.689616 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 15:16:40.689147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:40.689111 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f5wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:16:40.689722 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:40.689691 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 15:16:41.095855 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:16:41.095759 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:16:41.693401 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:41.693364 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f5wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:16:41.694093 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:41.694059 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 15:16:42.690020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:42.689437 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f5wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:16:42.690020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:42.689526 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 15:16:43.689238 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:43.689193 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f5wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:16:43.689640 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:43.689267 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 15:16:43.923244 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:43.923204 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:44.689889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:44.689850 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-f5wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 15:16:44.690321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:44.689916 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 15:16:45.912984 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:45.912953 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:45.913536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:45.913514 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:45.914383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:45.914360 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:16:49.074787 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:49.074751 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:50.648294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:50.648254 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:54.165601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:54.165563 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:16:59.385370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:16:59.385332 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:04.611243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:04.611208 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:09.669373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:09.669331 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:11.011123 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:17:11.011084 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:17:14.674367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:14.674307 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:18.763139 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.763105 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78"] Nov 02 15:17:18.767580 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.767561 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.849229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.849190 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:18.862338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.862294 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"distinct-w5-891723-kserve-self-signed-certs\"" Nov 02 15:17:18.864360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.864340 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78"] Nov 02 15:17:18.868037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.868018 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tmp-dir\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.868109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.868056 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-dshm\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.868109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.868088 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-home\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.868229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.868207 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skfd4\" (UniqueName: \"kubernetes.io/projected/867c049b-fd84-4fe1-9e9a-939b8e387b8f-kube-api-access-skfd4\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.868326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.868313 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tls-certs\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.868389 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.868379 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-model-cache\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.969031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.968995 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tls-certs\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.969184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.969092 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-model-cache\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.969184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.969125 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tmp-dir\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.969184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.969156 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-dshm\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.969355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.969196 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-home\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.969355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.969222 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skfd4\" (UniqueName: \"kubernetes.io/projected/867c049b-fd84-4fe1-9e9a-939b8e387b8f-kube-api-access-skfd4\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.969530 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.969508 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-model-cache\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.969599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.969545 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-home\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.969779 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.969761 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tmp-dir\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.971448 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.971418 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-dshm\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:18.971676 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:18.971658 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tls-certs\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:19.028471 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:19.028406 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skfd4\" (UniqueName: \"kubernetes.io/projected/867c049b-fd84-4fe1-9e9a-939b8e387b8f-kube-api-access-skfd4\") pod \"distinct-w5-891723-kserve-675d8c7744-dlk78\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:19.078946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:19.078914 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:19.486360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:19.486331 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78"] Nov 02 15:17:19.488955 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:17:19.488922 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod867c049b_fd84_4fe1_9e9a_939b8e387b8f.slice/crio-60f379b8ed59b34ead41995a45cb57a89457ab90ec83bc1e2dd808b01c033607 WatchSource:0}: Error finding container 60f379b8ed59b34ead41995a45cb57a89457ab90ec83bc1e2dd808b01c033607: Status 404 returned error can't find the container with id 60f379b8ed59b34ead41995a45cb57a89457ab90ec83bc1e2dd808b01c033607 Nov 02 15:17:19.841110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:19.841014 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" event={"ID":"867c049b-fd84-4fe1-9e9a-939b8e387b8f","Type":"ContainerStarted","Data":"057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9"} Nov 02 15:17:19.841110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:19.841064 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" event={"ID":"867c049b-fd84-4fe1-9e9a-939b8e387b8f","Type":"ContainerStarted","Data":"60f379b8ed59b34ead41995a45cb57a89457ab90ec83bc1e2dd808b01c033607"} Nov 02 15:17:19.867684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:19.867647 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:19.956778 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:19.956679 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podStartSLOduration=1.956660473 podStartE2EDuration="1.956660473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:19.956044857 +0000 UTC m=+1143.629248052" watchObservedRunningTime="2026-11-02 15:17:19.956660473 +0000 UTC m=+1143.629863668" Nov 02 15:17:20.081138 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:20.081105 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:22.149881 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:22.149850 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:24.869373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:24.869335 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:25.651991 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.651933 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq"] Nov 02 15:17:25.655948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.655928 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.755935 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.755901 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq"] Nov 02 15:17:25.772843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.772793 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.772843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.772835 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.773159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.772924 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.773159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.772951 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.773159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.773049 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv6bx\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-kube-api-access-kv6bx\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.773386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.773155 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.773386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.773215 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435423ae-48db-4912-9d55-5bc14cbd7fbf-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.773386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.773317 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.773386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.773362 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.874661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.874630 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435423ae-48db-4912-9d55-5bc14cbd7fbf-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.874719 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.874774 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.874822 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.874856 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.874910 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.874938 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.875082 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv6bx\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-kube-api-access-kv6bx\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.875145 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.875557 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.875607 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.875648 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.875696 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.875921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.875885 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435423ae-48db-4912-9d55-5bc14cbd7fbf-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.877427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.877404 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.877517 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.877448 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.932927 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.932862 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.933069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.933033 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv6bx\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-kube-api-access-kv6bx\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:25.970012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:25.969969 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:26.392228 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:26.392197 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq"] Nov 02 15:17:26.393449 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:17:26.393421 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435423ae_48db_4912_9d55_5bc14cbd7fbf.slice/crio-2376ad880814c685662bcbfa41d0351047d8e68a6d76e1c00b868995a0a92cad WatchSource:0}: Error finding container 2376ad880814c685662bcbfa41d0351047d8e68a6d76e1c00b868995a0a92cad: Status 404 returned error can't find the container with id 2376ad880814c685662bcbfa41d0351047d8e68a6d76e1c00b868995a0a92cad Nov 02 15:17:26.395869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:26.395833 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:17:26.395952 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:26.395909 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:17:26.395952 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:26.395937 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:17:26.884831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:26.884799 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" event={"ID":"435423ae-48db-4912-9d55-5bc14cbd7fbf","Type":"ContainerStarted","Data":"7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906"} Nov 02 15:17:26.885321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:26.884851 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" event={"ID":"435423ae-48db-4912-9d55-5bc14cbd7fbf","Type":"ContainerStarted","Data":"2376ad880814c685662bcbfa41d0351047d8e68a6d76e1c00b868995a0a92cad"} Nov 02 15:17:26.970404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:26.970375 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:27.000324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.000264 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" podStartSLOduration=2.000246956 podStartE2EDuration="2.000246956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:26.999470554 +0000 UTC m=+1150.672673748" watchObservedRunningTime="2026-11-02 15:17:27.000246956 +0000 UTC m=+1150.673450173" Nov 02 15:17:27.131994 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.131953 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r"] Nov 02 15:17:27.135824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.135777 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.184761 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.184727 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-61a835-openshift-default-dockercfg-pq94w\"" Nov 02 15:17:27.201694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.201658 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r"] Nov 02 15:17:27.291724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.291690 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.291724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.291740 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.291937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.291761 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.291937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.291831 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.291937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.291923 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.292070 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.292002 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2qjg\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-kube-api-access-q2qjg\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.292140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.292124 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c01092-3876-411c-80f1-f1edb9e90f2b-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.292177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.292159 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.292214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.292189 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393147 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393198 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393316 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393368 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393452 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2qjg\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-kube-api-access-q2qjg\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393600 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c01092-3876-411c-80f1-f1edb9e90f2b-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393614 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393639 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393685 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393696 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.393804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393766 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.394010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.393936 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.394180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.394156 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.394482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.394459 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c01092-3876-411c-80f1-f1edb9e90f2b-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.395783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.395758 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.396020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.396004 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.452222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.452187 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.452378 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.452235 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2qjg\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-kube-api-access-q2qjg\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.750050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.749951 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:27.972452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.972418 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 15:17:27.972802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:27.972474 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 15:17:28.169613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:28.169581 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r"] Nov 02 15:17:28.171486 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:17:28.171445 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c01092_3876_411c_80f1_f1edb9e90f2b.slice/crio-c851b560f117c72f22abdbfe49a050fc4fd58d94bcd9e3c1daf4f60bb61f3c76 WatchSource:0}: Error finding container c851b560f117c72f22abdbfe49a050fc4fd58d94bcd9e3c1daf4f60bb61f3c76: Status 404 returned error can't find the container with id c851b560f117c72f22abdbfe49a050fc4fd58d94bcd9e3c1daf4f60bb61f3c76 Nov 02 15:17:28.173917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:28.173886 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:17:28.174028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:28.173946 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:17:28.174075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:28.174026 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:17:28.897239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:28.896691 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" event={"ID":"25c01092-3876-411c-80f1-f1edb9e90f2b","Type":"ContainerStarted","Data":"c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46"} Nov 02 15:17:28.897239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:28.896750 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" event={"ID":"25c01092-3876-411c-80f1-f1edb9e90f2b","Type":"ContainerStarted","Data":"c851b560f117c72f22abdbfe49a050fc4fd58d94bcd9e3c1daf4f60bb61f3c76"} Nov 02 15:17:28.971009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:28.970954 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 15:17:28.971175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:28.971033 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 15:17:29.084996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:29.084946 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:29.200965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:29.200873 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:29.336004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:29.335738 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" podStartSLOduration=2.3357141710000002 podStartE2EDuration="2.335714171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:29.03164551 +0000 UTC m=+1152.704848706" watchObservedRunningTime="2026-11-02 15:17:29.335714171 +0000 UTC m=+1153.008917367" Nov 02 15:17:29.759897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:29.759862 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:29.771399 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:29.771366 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:17:29.771643 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:29.771616 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 15:17:29.988000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:29.979219 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 15:17:29.988000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:29.979296 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 15:17:29.990920 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:29.990889 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:17:30.188001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:30.187433 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:30.751418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:30.751384 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:17:30.751675 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:30.751646 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 15:17:30.970783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:30.970735 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 15:17:30.970990 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:30.970814 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 15:17:31.751022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:31.750421 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:17:31.751022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:31.750488 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 15:17:31.971066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:31.971024 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 15:17:31.971270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:31.971086 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 15:17:32.750468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:32.750426 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:17:32.750769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:32.750731 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 15:17:32.970853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:32.970820 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 15:17:32.971273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:32.970884 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 15:17:33.750817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:33.750781 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:17:33.751020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:33.750835 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 15:17:33.974827 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:33.974796 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:33.975276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:33.975180 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:33.975828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:33.975808 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:17:34.750415 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:34.750381 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 15:17:34.750593 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:34.750442 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 15:17:35.577574 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:35.577531 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:35.838102 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:35.837998 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:35.840169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:35.840138 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:35.845908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:35.845883 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:17:37.946265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:37.946232 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl"] Nov 02 15:17:37.949822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:37.949804 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:37.988636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:37.988604 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tls-certs\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:37.988636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:37.988640 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tmp-dir\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:37.988883 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:37.988769 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-dshm\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:37.988883 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:37.988821 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-home\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:37.989013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:37.988891 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfcvp\" (UniqueName: \"kubernetes.io/projected/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-kube-api-access-nfcvp\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:37.989013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:37.988937 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-model-cache\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.038173 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.038140 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:38.045007 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.044967 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"distinct-w6-e98976-kserve-self-signed-certs\"" Nov 02 15:17:38.047468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.047445 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl"] Nov 02 15:17:38.109954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.109915 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tls-certs\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.110319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.110220 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tmp-dir\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.110582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.110563 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-dshm\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.110845 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.110827 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-home\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.110969 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.110953 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcvp\" (UniqueName: \"kubernetes.io/projected/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-kube-api-access-nfcvp\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.111234 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.111209 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-model-cache\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.112107 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.112083 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-model-cache\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.112485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.112467 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-home\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.112889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.112849 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tmp-dir\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.123391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.122593 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-dshm\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.126711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.126681 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tls-certs\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.170513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.170479 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcvp\" (UniqueName: \"kubernetes.io/projected/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-kube-api-access-nfcvp\") pod \"distinct-w6-e98976-kserve-866b4b8767-t77cl\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.261320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.261237 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:38.716918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.716893 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl"] Nov 02 15:17:38.719112 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:17:38.719076 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa5f6ccc_64fe_4d23_b350_ea81c5ea7ba9.slice/crio-e0775e6535e76a4a696af6374cbdf877c5174ea03e3fbc52a638c4b7263cf7a3 WatchSource:0}: Error finding container e0775e6535e76a4a696af6374cbdf877c5174ea03e3fbc52a638c4b7263cf7a3: Status 404 returned error can't find the container with id e0775e6535e76a4a696af6374cbdf877c5174ea03e3fbc52a638c4b7263cf7a3 Nov 02 15:17:38.981162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.981073 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" event={"ID":"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9","Type":"ContainerStarted","Data":"260cbda54fd6ca76b62537ce32e3533da036eb7f5a2f9ed96e06088e1302c29a"} Nov 02 15:17:38.981162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:38.981115 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" event={"ID":"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9","Type":"ContainerStarted","Data":"e0775e6535e76a4a696af6374cbdf877c5174ea03e3fbc52a638c4b7263cf7a3"} Nov 02 15:17:39.091802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:39.091739 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podStartSLOduration=2.0917216 podStartE2EDuration="2.0917216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:39.091049096 +0000 UTC m=+1162.764252292" watchObservedRunningTime="2026-11-02 15:17:39.0917216 +0000 UTC m=+1162.764924792" Nov 02 15:17:39.262313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:39.262239 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:40.625673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:40.625635 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:41.012838 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:17:41.012746 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:17:45.372055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:45.371934 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:45.727860 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:45.727825 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:48.261698 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:48.261661 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:48.274420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:48.274392 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:49.048808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:49.048779 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:17:50.773023 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:50.772962 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:51.221825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:51.221791 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:17:55.966769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:17:55.966728 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:00.979590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:00.979560 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:01.127645 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:01.127613 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:04.735882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:04.735849 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:06.081360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:06.081307 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:10.727724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.727691 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl"] Nov 02 15:18:10.731662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.731640 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.815092 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.815059 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.815274 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.815127 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.815274 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.815198 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.815353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.815296 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.815353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.815314 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.815353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.815333 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.815485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.815450 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.815542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.815526 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bwv\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-kube-api-access-n2bwv\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.815583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.815566 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668ed6ff-847a-4991-847f-42f45b7bd7d8-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.831805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.831771 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl"] Nov 02 15:18:10.916243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916202 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bwv\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-kube-api-access-n2bwv\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.916410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916256 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668ed6ff-847a-4991-847f-42f45b7bd7d8-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.916410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916308 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.916410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916357 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.916410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916381 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.916572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916447 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.916572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916472 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.916572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916500 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.916572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916562 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.916912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916852 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.917089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.916922 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.917089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.917078 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.917215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.917185 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668ed6ff-847a-4991-847f-42f45b7bd7d8-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.917339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.917316 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.919020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.918989 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.919216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.919200 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.974133 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.974101 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:10.974278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:10.974212 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bwv\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-kube-api-access-n2bwv\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:11.012072 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:18:11.011961 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:18:11.046233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:11.046201 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:11.175218 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:11.175183 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:11.472988 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:11.472943 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl"] Nov 02 15:18:11.474594 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:18:11.474563 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668ed6ff_847a_4991_847f_42f45b7bd7d8.slice/crio-6d289ee10907f4259cb612e1d5e08f35b0eea52043f0fc5d0607531bcfaf7275 WatchSource:0}: Error finding container 6d289ee10907f4259cb612e1d5e08f35b0eea52043f0fc5d0607531bcfaf7275: Status 404 returned error can't find the container with id 6d289ee10907f4259cb612e1d5e08f35b0eea52043f0fc5d0607531bcfaf7275 Nov 02 15:18:11.477035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:11.477003 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:11.477116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:11.477070 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:11.477116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:11.477101 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:12.177740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:12.177706 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" event={"ID":"668ed6ff-847a-4991-847f-42f45b7bd7d8","Type":"ContainerStarted","Data":"4bfea21187a8f34216cb5e3c54009833c57f2993fb67baf4486c1dbadeb6a1cf"} Nov 02 15:18:12.178170 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:12.177763 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" event={"ID":"668ed6ff-847a-4991-847f-42f45b7bd7d8","Type":"ContainerStarted","Data":"6d289ee10907f4259cb612e1d5e08f35b0eea52043f0fc5d0607531bcfaf7275"} Nov 02 15:18:12.298999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:12.295293 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podStartSLOduration=2.295269704 podStartE2EDuration="2.295269704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:12.293129179 +0000 UTC m=+1195.966332374" watchObservedRunningTime="2026-11-02 15:18:12.295269704 +0000 UTC m=+1195.968472931" Nov 02 15:18:13.046770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:13.046729 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:13.052147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:13.052117 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:18:13.052292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:13.052167 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 15:18:14.046909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:14.046870 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:18:14.047356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:14.046927 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 15:18:14.453527 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:14.453492 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:15.047463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:15.047427 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:18:15.047887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:15.047499 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 15:18:15.261991 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:15.261937 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:16.048995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:16.048269 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:18:16.048995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:16.048327 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 15:18:16.418902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:16.418839 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:17.050378 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:17.050337 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:18:17.051005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:17.050951 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 15:18:18.046693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:18.046647 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 15:18:18.046878 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:18.046718 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 15:18:19.051947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:19.051917 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:19.052345 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:19.052294 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:19.053050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:19.053033 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:18:21.598206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:21.598165 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:22.476658 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:22.476613 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:26.677089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:26.677052 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:27.204194 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.204156 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w"] Nov 02 15:18:27.209566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.209542 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.258373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.258340 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"kube-root-ca.crt\"" Nov 02 15:18:27.302192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.302160 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tmp-dir\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.302354 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.302214 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-dshm\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.302354 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.302333 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tls-certs\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.302425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.302382 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-home\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.302467 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.302432 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdvrx\" (UniqueName: \"kubernetes.io/projected/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-kube-api-access-mdvrx\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.302501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.302465 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-model-cache\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.304520 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.304496 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"openshift-service-ca.crt\"" Nov 02 15:18:27.304695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.304594 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"distinct-w0-61a835-kserve-self-signed-certs\"" Nov 02 15:18:27.304695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.304622 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"default-dockercfg-bxzxs\"" Nov 02 15:18:27.307342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.307319 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w"] Nov 02 15:18:27.321336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.321309 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:27.405571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.404172 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tmp-dir\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.405571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.404267 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-dshm\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.405571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.404356 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tls-certs\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.405571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.404401 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-home\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.405571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.404425 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdvrx\" (UniqueName: \"kubernetes.io/projected/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-kube-api-access-mdvrx\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.405571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.404467 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-model-cache\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.405571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.404964 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-model-cache\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.405571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.405250 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tmp-dir\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.406159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.405942 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-home\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.421104 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.413120 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tls-certs\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.430766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.430733 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-dshm\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.474595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.474517 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdvrx\" (UniqueName: \"kubernetes.io/projected/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-kube-api-access-mdvrx\") pod \"distinct-w0-61a835-kserve-66b4c796cc-cgq9w\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.522576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.522542 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:27.917271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:27.917242 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w"] Nov 02 15:18:27.919516 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:18:27.919487 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37cb0552_aaa7_444b_b8cf_cc1eb3b57546.slice/crio-fbf07ba2d68414be90e2e4f471261e80c93da0209042280bacec2dc6dfb6dc56 WatchSource:0}: Error finding container fbf07ba2d68414be90e2e4f471261e80c93da0209042280bacec2dc6dfb6dc56: Status 404 returned error can't find the container with id fbf07ba2d68414be90e2e4f471261e80c93da0209042280bacec2dc6dfb6dc56 Nov 02 15:18:28.338410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:28.338326 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" event={"ID":"37cb0552-aaa7-444b-b8cf-cc1eb3b57546","Type":"ContainerStarted","Data":"c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7"} Nov 02 15:18:28.338410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:28.338366 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" event={"ID":"37cb0552-aaa7-444b-b8cf-cc1eb3b57546","Type":"ContainerStarted","Data":"fbf07ba2d68414be90e2e4f471261e80c93da0209042280bacec2dc6dfb6dc56"} Nov 02 15:18:28.453332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:28.453275 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podStartSLOduration=1.453255648 podStartE2EDuration="1.453255648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:28.450303392 +0000 UTC m=+1212.123506587" watchObservedRunningTime="2026-11-02 15:18:28.453255648 +0000 UTC m=+1212.126458847" Nov 02 15:18:28.523363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:28.523326 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:31.868173 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:31.868129 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:36.864811 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:36.864774 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:37.522950 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:37.522917 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:37.535475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:37.535447 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:38.406792 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:38.406763 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:18:41.012253 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:18:41.012206 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:18:41.611035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.611002 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj"] Nov 02 15:18:41.614185 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.614168 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.687875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.687834 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.688068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.688005 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq9hm\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-kube-api-access-vq9hm\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.688068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.688034 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.688158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.688117 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2957d-3b26-49a9-862c-6ff5d83cb626-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.688190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.688164 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.688220 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.688185 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.688325 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.688312 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.688359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.688337 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.688396 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.688362 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.714496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.714464 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj"] Nov 02 15:18:41.789796 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.789762 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.789804 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.789853 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.789880 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.789922 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.790101 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.790248 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq9hm\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-kube-api-access-vq9hm\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.790285 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.790295 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.790319 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790557 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.790385 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2957d-3b26-49a9-862c-6ff5d83cb626-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790557 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.790399 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.790639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.790567 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.791181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.791160 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2957d-3b26-49a9-862c-6ff5d83cb626-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.792407 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.792383 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.792541 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.792523 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.847096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.847063 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.847248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.847202 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq9hm\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-kube-api-access-vq9hm\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:41.928304 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:41.928265 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:42.075311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.075279 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp"] Nov 02 15:18:42.079074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.079054 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.097277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.097252 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.097417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.097291 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.097417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.097309 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.097417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.097324 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.097554 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.097479 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn7nv\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-kube-api-access-tn7nv\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.097554 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.097539 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.097630 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.097609 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.097686 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.097640 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.097744 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.097730 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.174334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.174229 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:42.188049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.184473 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp"] Nov 02 15:18:42.200079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.200019 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7nv\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-kube-api-access-tn7nv\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.200356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.200330 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.200555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.200537 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.200680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.200667 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.200803 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.200789 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.201038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.201023 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.201815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.201152 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.201998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.201958 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.202123 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.202108 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.202921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.202897 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.203076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.201072 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.203523 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.203503 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.203652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.201307 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.206510 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.206479 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.206686 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.201539 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.208164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.207511 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.259334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.259304 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.259490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.259403 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7nv\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-kube-api-access-tn7nv\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.370373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.370349 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj"] Nov 02 15:18:42.371714 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:18:42.371687 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a2957d_3b26_49a9_862c_6ff5d83cb626.slice/crio-98f9b889bf711da9557e6c5e6949a28977f5c1fe8f1e19ab88c27bc000a4395e WatchSource:0}: Error finding container 98f9b889bf711da9557e6c5e6949a28977f5c1fe8f1e19ab88c27bc000a4395e: Status 404 returned error can't find the container with id 98f9b889bf711da9557e6c5e6949a28977f5c1fe8f1e19ab88c27bc000a4395e Nov 02 15:18:42.374250 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.374221 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:42.374329 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.374286 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:42.374329 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.374319 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:42.392500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.392480 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:42.421237 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.421205 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" event={"ID":"92a2957d-3b26-49a9-862c-6ff5d83cb626","Type":"ContainerStarted","Data":"98f9b889bf711da9557e6c5e6949a28977f5c1fe8f1e19ab88c27bc000a4395e"} Nov 02 15:18:42.823585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.823546 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp"] Nov 02 15:18:42.825303 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:18:42.825265 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0280aa67_0b9a_4ae7_a3b6_2efc3b875fc4.slice/crio-07c9fc05618e3be896233de7aa2161bb230fff4aafdf431be9b4c8667d4c449c WatchSource:0}: Error finding container 07c9fc05618e3be896233de7aa2161bb230fff4aafdf431be9b4c8667d4c449c: Status 404 returned error can't find the container with id 07c9fc05618e3be896233de7aa2161bb230fff4aafdf431be9b4c8667d4c449c Nov 02 15:18:42.828446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.828407 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:42.828576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.828497 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:42.828576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:42.828545 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:43.438997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:43.438484 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" event={"ID":"92a2957d-3b26-49a9-862c-6ff5d83cb626","Type":"ContainerStarted","Data":"1ed2b2efdc468e66a9628e034a3be9e9a578c7db31bf8a431ce7d7bf488b04cb"} Nov 02 15:18:43.441696 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:43.441669 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" event={"ID":"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4","Type":"ContainerStarted","Data":"accc126df94036061a6f1ba101f39c9aa59fccd641b3a4fc476d7af88df06fbe"} Nov 02 15:18:43.441821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:43.441720 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" event={"ID":"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4","Type":"ContainerStarted","Data":"07c9fc05618e3be896233de7aa2161bb230fff4aafdf431be9b4c8667d4c449c"} Nov 02 15:18:43.554383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:43.554331 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" podStartSLOduration=2.554315238 podStartE2EDuration="2.554315238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:43.55207596 +0000 UTC m=+1227.225279157" watchObservedRunningTime="2026-11-02 15:18:43.554315238 +0000 UTC m=+1227.227518433" Nov 02 15:18:43.667875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:43.667820 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" podStartSLOduration=1.667804679 podStartE2EDuration="1.667804679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:43.665535186 +0000 UTC m=+1227.338738381" watchObservedRunningTime="2026-11-02 15:18:43.667804679 +0000 UTC m=+1227.341007878" Nov 02 15:18:43.938996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:43.938753 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:43.938996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:43.938896 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:18:43.938996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:43.938960 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 15:18:44.393512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:44.393478 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:44.394715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:44.394689 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:18:44.394835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:44.394737 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 15:18:44.938000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:44.932368 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:18:44.938000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:44.932429 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 15:18:45.393058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:45.393022 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:18:45.393231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:45.393080 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 15:18:45.941999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:45.941086 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:18:45.941999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:45.941153 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 15:18:46.393751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:46.393718 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:18:46.393919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:46.393780 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 15:18:46.946157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:46.946109 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:18:46.947422 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:46.947379 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 15:18:47.393383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:47.393344 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:18:47.393586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:47.393422 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 15:18:47.396302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:47.396257 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:47.936850 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:47.936383 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:18:47.936850 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:47.936461 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 15:18:48.393671 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:48.393639 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:18:48.394113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:48.393702 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 15:18:48.929739 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:48.929699 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:18:48.930068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:48.930034 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 15:18:49.393140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:49.393099 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:18:49.393353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:49.393180 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 15:18:50.311728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:50.311694 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:50.312243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:50.312142 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:50.313129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:50.313107 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:18:50.397192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:50.397164 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:50.397457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:50.397435 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:50.398068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:50.398044 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:18:52.586414 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:52.586379 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:18:57.675319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:18:57.675285 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:02.762467 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:02.762430 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:07.854272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:07.854235 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:11.010470 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:19:11.010428 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:19:13.045043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.044962 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf"] Nov 02 15:19:13.054271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.054242 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.061426 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.061353 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:13.090886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.090802 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.091131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.091059 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpvb\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-kube-api-access-tjpvb\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.092122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.091395 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038259bc-614d-48df-b46b-33544964ee15-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.092122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.091451 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.092122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.091625 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.092122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.091654 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038259bc-614d-48df-b46b-33544964ee15-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.092122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.091712 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.092122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.091783 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.092122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.091816 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.097631 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.097601 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf"] Nov 02 15:19:13.192872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.192836 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.192872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.192872 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038259bc-614d-48df-b46b-33544964ee15-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.192893 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.192920 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.192951 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.193076 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.193226 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpvb\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-kube-api-access-tjpvb\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.193279 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038259bc-614d-48df-b46b-33544964ee15-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.193321 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.193327 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.193425 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.193500 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.193740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.193719 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.194184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.194160 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038259bc-614d-48df-b46b-33544964ee15-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.195378 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.195355 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.195500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.195429 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038259bc-614d-48df-b46b-33544964ee15-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.250230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.250198 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.250478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.250456 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpvb\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-kube-api-access-tjpvb\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.383929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.383899 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:13.801186 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.801159 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf"] Nov 02 15:19:13.802576 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:19:13.802542 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038259bc_614d_48df_b46b_33544964ee15.slice/crio-10d73549b90da5ca5d2ed67769914b3e44ba0adab403bb756ed6f845deb71eb0 WatchSource:0}: Error finding container 10d73549b90da5ca5d2ed67769914b3e44ba0adab403bb756ed6f845deb71eb0: Status 404 returned error can't find the container with id 10d73549b90da5ca5d2ed67769914b3e44ba0adab403bb756ed6f845deb71eb0 Nov 02 15:19:13.804804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.804768 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:19:13.804912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.804856 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:19:13.805005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:13.804907 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:19:14.749819 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:14.749785 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" event={"ID":"038259bc-614d-48df-b46b-33544964ee15","Type":"ContainerStarted","Data":"fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d"} Nov 02 15:19:14.750277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:14.749851 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" event={"ID":"038259bc-614d-48df-b46b-33544964ee15","Type":"ContainerStarted","Data":"10d73549b90da5ca5d2ed67769914b3e44ba0adab403bb756ed6f845deb71eb0"} Nov 02 15:19:14.913813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:14.913751 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" podStartSLOduration=2.913735751 podStartE2EDuration="2.913735751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:14.91087181 +0000 UTC m=+1258.584075005" watchObservedRunningTime="2026-11-02 15:19:14.913735751 +0000 UTC m=+1258.586938945" Nov 02 15:19:15.384714 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:15.384675 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:15.386194 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:15.386167 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 15:19:15.386321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:15.386216 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 15:19:15.742779 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:15.742687 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:16.385126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:16.385058 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 15:19:16.385126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:16.385113 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 15:19:17.384912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:17.384879 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 15:19:17.385111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:17.384934 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 15:19:18.205925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:18.205883 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:18.384504 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:18.384471 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 15:19:18.384683 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:18.384527 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 15:19:19.384952 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:19.384915 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 15:19:19.385465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:19.384970 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 15:19:20.384207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:20.384171 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 15:19:20.384377 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:20.384221 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 15:19:21.389320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:21.389288 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:21.389740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:21.389717 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:21.390399 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:21.390377 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:19:22.843114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:22.843084 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:23.368647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:23.368615 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:28.630438 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:28.630355 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:31.343376 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:31.343331 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:31.497239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:31.497200 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:33.874762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:33.874725 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:35.658970 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:35.658933 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:35.808096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:35.808056 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:39.070146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:39.070109 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:41.012244 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:19:41.012205 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:19:44.065479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:44.065445 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:44.239105 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:44.239065 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:49.485261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:49.485229 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:54.685577 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:54.685542 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:19:59.882859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:19:59.882825 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:01.811869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:01.811731 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:05.218352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:05.218314 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:07.538271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:07.538242 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:10.472141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:10.472106 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:11.010511 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:20:11.010474 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:20:15.659227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:15.659193 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:16.261799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:16.261761 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:17.349443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.349411 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk"] Nov 02 15:20:17.353312 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.353295 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.379816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.379789 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdg9d\" (UniqueName: \"kubernetes.io/projected/1972a793-bbcf-407c-86d1-bfaa67eddcb8-kube-api-access-mdg9d\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.379957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.379855 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-home\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.379957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.379952 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-dshm\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.380112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.380000 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-model-cache\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.380112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.380036 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tmp-dir\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.380112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.380051 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tls-certs\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.403385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.403352 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"distinct-2-w3-71d1eb-kserve-self-signed-certs\"" Nov 02 15:20:17.403527 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.403389 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"openshift-service-ca.crt\"" Nov 02 15:20:17.419488 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.419467 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"default-dockercfg-z5s42\"" Nov 02 15:20:17.449421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.449398 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"kube-root-ca.crt\"" Nov 02 15:20:17.452077 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.452059 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk"] Nov 02 15:20:17.453443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.453420 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:17.481216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.481183 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-home\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.481334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.481283 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-dshm\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.481334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.481309 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-model-cache\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.481445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.481359 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tmp-dir\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.481445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.481386 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tls-certs\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.481574 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.481548 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdg9d\" (UniqueName: \"kubernetes.io/projected/1972a793-bbcf-407c-86d1-bfaa67eddcb8-kube-api-access-mdg9d\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.481632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.481592 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-home\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.481699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.481672 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tmp-dir\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.481802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.481775 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-model-cache\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.483486 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.483468 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-dshm\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.483907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.483890 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tls-certs\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.542332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.542267 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdg9d\" (UniqueName: \"kubernetes.io/projected/1972a793-bbcf-407c-86d1-bfaa67eddcb8-kube-api-access-mdg9d\") pod \"distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:17.665710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:17.665675 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:18.137179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:18.137151 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk"] Nov 02 15:20:18.139195 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:20:18.139166 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1972a793_bbcf_407c_86d1_bfaa67eddcb8.slice/crio-b83d5053d69513b220d9801afbe521a47146175f70e9e227ca82ef8fdde6a9cb WatchSource:0}: Error finding container b83d5053d69513b220d9801afbe521a47146175f70e9e227ca82ef8fdde6a9cb: Status 404 returned error can't find the container with id b83d5053d69513b220d9801afbe521a47146175f70e9e227ca82ef8fdde6a9cb Nov 02 15:20:18.222656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:18.222623 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" event={"ID":"1972a793-bbcf-407c-86d1-bfaa67eddcb8","Type":"ContainerStarted","Data":"9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629"} Nov 02 15:20:18.222801 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:18.222672 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" event={"ID":"1972a793-bbcf-407c-86d1-bfaa67eddcb8","Type":"ContainerStarted","Data":"b83d5053d69513b220d9801afbe521a47146175f70e9e227ca82ef8fdde6a9cb"} Nov 02 15:20:18.345296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:18.345244 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podStartSLOduration=1.345229738 podStartE2EDuration="1.345229738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:18.342656573 +0000 UTC m=+1322.015859771" watchObservedRunningTime="2026-11-02 15:20:18.345229738 +0000 UTC m=+1322.018432932" Nov 02 15:20:18.666747 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:18.666699 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:20.886395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:20.886359 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:22.025669 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:22.025618 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:23.142406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:23.142373 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:26.051125 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:26.051082 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:27.470946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.470908 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn"] Nov 02 15:20:27.475336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.475312 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.499792 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.499759 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m"] Nov 02 15:20:27.503865 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.503844 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.575439 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.575401 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn"] Nov 02 15:20:27.578633 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.578598 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m"] Nov 02 15:20:27.614379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.614338 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.614560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.614462 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.614560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.614506 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.614670 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.614643 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.614704 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.614689 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.614799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.614780 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.614833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.614810 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.614833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.614827 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.615021 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615001 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.615060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615039 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6f4\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-kube-api-access-9m6f4\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.615126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615106 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.615203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615187 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.615239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615225 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.615277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615250 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f9d780-9ea7-4084-8636-b99d44a79c3f-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.615312 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615285 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.615381 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615366 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.615449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615435 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.615483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.615471 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxbg\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-kube-api-access-mmxbg\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.666087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.666050 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:27.679563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.679535 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:27.717148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717115 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.717148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717152 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.717411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717222 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.717411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717241 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6f4\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-kube-api-access-9m6f4\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.717411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717265 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.717411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717293 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.717411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717323 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.717411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717349 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f9d780-9ea7-4084-8636-b99d44a79c3f-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.717411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717375 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.717775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717460 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.717775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717544 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.717775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717583 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxbg\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-kube-api-access-mmxbg\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.717775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717585 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.717775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717587 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.718051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717806 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.718051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717887 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.718051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717924 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.718051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.717940 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.718051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718024 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.718051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718031 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.718265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718116 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.718265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718153 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.718265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718235 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.718265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718245 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.718407 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718294 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f9d780-9ea7-4084-8636-b99d44a79c3f-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.718407 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718343 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.718407 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718395 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.718877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.718851 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.720331 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.720312 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.720639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.720619 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.721110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.721061 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.721110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.721084 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.793513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.793483 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.793697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.793577 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxbg\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-kube-api-access-mmxbg\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:27.794287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.794260 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6f4\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-kube-api-access-9m6f4\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.801628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.801603 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.817257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.817231 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:27.828462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:27.828429 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:28.090180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.090094 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:28.245183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.245157 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m"] Nov 02 15:20:28.247489 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:20:28.247455 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f9d780_9ea7_4084_8636_b99d44a79c3f.slice/crio-2e1cabf63918efd3633b163ff44ea057a67d2cb2400875899b01d1ac3217e226 WatchSource:0}: Error finding container 2e1cabf63918efd3633b163ff44ea057a67d2cb2400875899b01d1ac3217e226: Status 404 returned error can't find the container with id 2e1cabf63918efd3633b163ff44ea057a67d2cb2400875899b01d1ac3217e226 Nov 02 15:20:28.249824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.249794 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.249900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.249858 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.249900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.249888 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.282084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.282049 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" event={"ID":"26f9d780-9ea7-4084-8636-b99d44a79c3f","Type":"ContainerStarted","Data":"2e1cabf63918efd3633b163ff44ea057a67d2cb2400875899b01d1ac3217e226"} Nov 02 15:20:28.295654 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.295626 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:28.482214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.482170 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn"] Nov 02 15:20:28.487224 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:20:28.487193 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cb1486_dc70_41b9_b3e8_6a25c6e9f8e1.slice/crio-e56edf1a86fdee79940fbd5c2b3421d45cc85d77e65c176e7db273c8d904bdf9 WatchSource:0}: Error finding container e56edf1a86fdee79940fbd5c2b3421d45cc85d77e65c176e7db273c8d904bdf9: Status 404 returned error can't find the container with id e56edf1a86fdee79940fbd5c2b3421d45cc85d77e65c176e7db273c8d904bdf9 Nov 02 15:20:28.490160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.490123 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.490237 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.490220 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.490278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:28.490263 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:29.290952 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:29.290920 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" event={"ID":"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1","Type":"ContainerStarted","Data":"99219287e538959f94e4b7f65fa2ca3da7c15596891f147a6db886fb93d5d336"} Nov 02 15:20:29.291242 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:29.291220 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" event={"ID":"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1","Type":"ContainerStarted","Data":"e56edf1a86fdee79940fbd5c2b3421d45cc85d77e65c176e7db273c8d904bdf9"} Nov 02 15:20:29.293519 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:29.293334 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" event={"ID":"26f9d780-9ea7-4084-8636-b99d44a79c3f","Type":"ContainerStarted","Data":"2ab6bee5b2a31ec4f0f84997b17c69e36301c1aaaa88f09d31eab9daed2a5c61"} Nov 02 15:20:29.670998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:29.669763 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podStartSLOduration=2.669744723 podStartE2EDuration="2.669744723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:29.528298671 +0000 UTC m=+1333.201501867" watchObservedRunningTime="2026-11-02 15:20:29.669744723 +0000 UTC m=+1333.342947917" Nov 02 15:20:29.670998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:29.670010 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podStartSLOduration=2.670002791 podStartE2EDuration="2.670002791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:29.667009642 +0000 UTC m=+1333.340212838" watchObservedRunningTime="2026-11-02 15:20:29.670002791 +0000 UTC m=+1333.343206066" Nov 02 15:20:29.834419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:29.834381 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:29.838137 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:29.838102 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 15:20:29.838465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:29.838421 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 15:20:30.090875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:30.090779 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:30.091936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:30.091913 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:20:30.092072 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:30.091960 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 15:20:30.840006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:30.822352 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 15:20:30.840006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:30.822411 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 15:20:31.104059 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:31.103490 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:20:31.104059 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:31.103564 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 15:20:31.823999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:31.820361 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 15:20:31.823999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:31.820427 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 15:20:32.000006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:31.998902 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:32.093112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.090114 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.096211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.096109 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:20:32.096211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.096171 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 15:20:32.315122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.315066 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:20:32.434001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.432105 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.820065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.819850 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 15:20:32.820065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.819913 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 15:20:32.874113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.869583 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:20:32.918001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.916160 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.918001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:32.916540 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:20:33.101069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.093288 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.101069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.093673 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:20:33.101069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.094798 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:20:33.101069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.094835 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 15:20:33.419123 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.413881 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.457998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.457095 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:33.473490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.473104 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:20:33.473490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.473416 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:20:33.712204 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.712093 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:20:33.830619 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.830576 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 15:20:33.830804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.830644 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 15:20:33.872124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.872071 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.000062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.917297 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:20:34.001701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.917334 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 15:20:34.019049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:33.986120 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:20:34.052000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.051085 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:20:34.052000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.051148 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:20:34.094096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.094050 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.162131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.156129 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:20:34.174000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.165093 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 15:20:34.418508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.418454 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.691086 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.675959 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:20:34.891000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.864512 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 15:20:34.891341 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.891302 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 15:20:34.994921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:34.994682 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:35.026161 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.026109 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:35.107382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.107330 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:35.107699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.107654 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:35.149008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.144460 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:20:35.149290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.149253 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 15:20:35.246082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.238087 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7"] Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.432268 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.443194 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.466565 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.467896 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-dshm\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.468022 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-tmp-dir\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.468083 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-home\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.468146 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/e9044377-0838-4232-81ba-1ea22536545e-kube-api-access-46qm8\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.468206 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9044377-0838-4232-81ba-1ea22536545e-tls-certs\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.468319 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-model-cache\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.542221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.512473 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7"] Nov 02 15:20:35.602030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.575526 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-tmp-dir\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.602030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.575596 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-home\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.602030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.575651 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/e9044377-0838-4232-81ba-1ea22536545e-kube-api-access-46qm8\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.602030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.575718 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9044377-0838-4232-81ba-1ea22536545e-tls-certs\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.602030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.575786 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-model-cache\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.602030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.575992 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-dshm\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.602030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.578773 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-tmp-dir\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.602030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.579048 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-home\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.602030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.579593 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-model-cache\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.628402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.604929 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"distinct-2-w0-61a835-kserve-self-signed-certs\"" Nov 02 15:20:35.703001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.685613 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9044377-0838-4232-81ba-1ea22536545e-tls-certs\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.710946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.702840 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-dshm\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.786614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.786587 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:20:35.787740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.787700 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/e9044377-0838-4232-81ba-1ea22536545e-kube-api-access-46qm8\") pod \"distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.797398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.797323 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:20:35.803908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.803868 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:35.814010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.812751 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:20:35.879386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.879308 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 15:20:35.879564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:35.879380 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 15:20:36.090758 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:36.090677 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:20:36.090758 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:36.090731 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 15:20:36.393048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:36.393016 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7"] Nov 02 15:20:36.822000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:36.820296 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 15:20:36.822000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:36.820357 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 15:20:36.894615 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:36.894575 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" event={"ID":"e9044377-0838-4232-81ba-1ea22536545e","Type":"ContainerStarted","Data":"61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed"} Nov 02 15:20:36.894786 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:36.894621 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" event={"ID":"e9044377-0838-4232-81ba-1ea22536545e","Type":"ContainerStarted","Data":"2185303dd784265f1e749a09c4b1cdb33c543c91cad7499c3c218803e04a80a6"} Nov 02 15:20:37.060638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:37.060548 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podStartSLOduration=3.060528189 podStartE2EDuration="3.060528189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:37.060186916 +0000 UTC m=+1340.733390131" watchObservedRunningTime="2026-11-02 15:20:37.060528189 +0000 UTC m=+1340.733731385" Nov 02 15:20:37.091144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:37.091059 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:20:37.091144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:37.091124 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 15:20:37.806870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:37.806840 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:37.818239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:37.818214 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 15:20:37.818373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:37.818263 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 15:20:38.091498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:38.091400 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 15:20:38.091498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:38.091464 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 15:20:38.302797 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:38.302733 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:39.825039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:39.824998 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:20:39.825445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:39.825072 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:39.826299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:39.826266 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:39.827146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:39.827128 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:39.900195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:39.900166 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:20:40.091236 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:40.091146 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:20:40.091475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:40.091443 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:40.150725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:40.150480 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:40.151583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:40.151551 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:40.153802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:40.153772 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:20:41.011368 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:20:41.011333 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:20:43.038363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.038329 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m"] Nov 02 15:20:43.042777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.042756 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.082049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.082011 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.082191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.082061 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.082254 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.082233 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.082291 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.082265 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.082351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.082337 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.082407 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.082397 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.082460 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.082448 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjl4\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-kube-api-access-jpjl4\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.082496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.082488 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.082548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.082536 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.142005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.141956 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m"] Nov 02 15:20:43.183863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.183822 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.183863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.183867 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.183917 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184029 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184058 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjl4\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-kube-api-access-jpjl4\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184112 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184151 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184277 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184319 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184489 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184397 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184545 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184507 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184668 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184778 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184707 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.184778 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.184725 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.186317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.186294 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.186512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.186493 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.241898 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.241862 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.242101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.241993 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjl4\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-kube-api-access-jpjl4\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.358735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.358655 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:43.783398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.783371 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m"] Nov 02 15:20:43.784472 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:20:43.784443 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51d10a9_c5d2_4026_ad91_7addeb1f1fa6.slice/crio-c3ce08e3008e619fc729c943b5310390a1fd1043344943f1e0b67066f82e136e WatchSource:0}: Error finding container c3ce08e3008e619fc729c943b5310390a1fd1043344943f1e0b67066f82e136e: Status 404 returned error can't find the container with id c3ce08e3008e619fc729c943b5310390a1fd1043344943f1e0b67066f82e136e Nov 02 15:20:43.786998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.786940 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:43.787090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.787022 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:43.787090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.787054 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:43.944284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.944244 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" event={"ID":"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6","Type":"ContainerStarted","Data":"382bb7dd39f2b8cdefde8ce6928738ee07cf99f2dd78f0af11a9af5f3ef81221"} Nov 02 15:20:43.944443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:43.944313 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" event={"ID":"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6","Type":"ContainerStarted","Data":"c3ce08e3008e619fc729c943b5310390a1fd1043344943f1e0b67066f82e136e"} Nov 02 15:20:44.065323 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.065219 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" podStartSLOduration=2.065204065 podStartE2EDuration="2.065204065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:44.062535361 +0000 UTC m=+1347.735738568" watchObservedRunningTime="2026-11-02 15:20:44.065204065 +0000 UTC m=+1347.738407259" Nov 02 15:20:44.172006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.171958 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j"] Nov 02 15:20:44.176189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.176169 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.197586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.197550 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.197725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.197631 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.197779 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.197722 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26cf8c8a-484a-42d7-b251-b4950a0c052a-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.197779 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.197768 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.197872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.197798 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.197872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.197848 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.197939 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.197882 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.198003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.197949 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.198068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.197967 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxj7\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-kube-api-access-5sxj7\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.250807 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.250773 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j"] Nov 02 15:20:44.299116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299075 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299123 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxj7\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-kube-api-access-5sxj7\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299309 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299395 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299432 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26cf8c8a-484a-42d7-b251-b4950a0c052a-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299492 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299518 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299572 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299609 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299878 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299848 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.299878 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299868 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.300071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.299956 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.300165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.300140 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.300582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.300556 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26cf8c8a-484a-42d7-b251-b4950a0c052a-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.302517 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.302494 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.302715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.302694 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.358929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.358840 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:44.360728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.360699 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:20:44.360891 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.360750 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 15:20:44.361771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.361747 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxj7\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-kube-api-access-5sxj7\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.362725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.362707 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.442526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.442488 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:44.490348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.490318 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:44.943653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.943626 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j"] Nov 02 15:20:44.946829 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:20:44.946803 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cf8c8a_484a_42d7_b251_b4950a0c052a.slice/crio-55acb86690ac1efc5dab31a274b07bdf3ec8ace95ccad559bb731b653279d616 WatchSource:0}: Error finding container 55acb86690ac1efc5dab31a274b07bdf3ec8ace95ccad559bb731b653279d616: Status 404 returned error can't find the container with id 55acb86690ac1efc5dab31a274b07bdf3ec8ace95ccad559bb731b653279d616 Nov 02 15:20:44.949382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.949356 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:44.949468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.949418 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:44.949468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:44.949449 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:45.359598 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:45.359514 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:20:45.359598 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:45.359579 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 15:20:45.806851 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:45.806811 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:45.873016 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:45.872951 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:45.967086 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:45.967046 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" event={"ID":"26cf8c8a-484a-42d7-b251-b4950a0c052a","Type":"ContainerStarted","Data":"35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3"} Nov 02 15:20:45.969344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:45.969314 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" event={"ID":"26cf8c8a-484a-42d7-b251-b4950a0c052a","Type":"ContainerStarted","Data":"55acb86690ac1efc5dab31a274b07bdf3ec8ace95ccad559bb731b653279d616"} Nov 02 15:20:45.982317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:45.982296 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:20:46.112916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:46.112809 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" podStartSLOduration=2.112794942 podStartE2EDuration="2.112794942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:46.108943397 +0000 UTC m=+1349.782146617" watchObservedRunningTime="2026-11-02 15:20:46.112794942 +0000 UTC m=+1349.785998136" Nov 02 15:20:46.365642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:46.365544 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:20:46.366654 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:46.366618 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 15:20:46.490659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:46.490625 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:46.491969 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:46.491947 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:20:46.492050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:46.492015 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 15:20:47.360998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:47.360294 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:20:47.360998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:47.360386 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 15:20:47.491295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:47.491257 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:20:47.491648 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:47.491313 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 15:20:48.360999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:48.360390 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:20:48.360999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:48.360457 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 15:20:48.491294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:48.491257 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:20:48.491459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:48.491313 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 15:20:48.794349 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:48.794312 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:49.362433 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:49.362395 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:20:49.362746 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:49.362709 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 15:20:49.491362 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:49.491327 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:20:49.491532 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:49.491378 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 15:20:50.359534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:50.359499 5433 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 15:20:50.359738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:50.359564 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 15:20:50.490825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:50.490793 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:20:50.491017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:50.490847 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 15:20:51.364359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:51.364326 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:51.364875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:51.364858 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:51.365625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:51.365601 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:20:51.491728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:51.491691 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 15:20:51.491900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:51.491759 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 15:20:52.495559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:52.495531 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:52.495947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:52.495826 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:52.496446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:52.496425 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:20:53.971568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:53.971533 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:57.337184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.337091 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn"] Nov 02 15:20:57.342554 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.342531 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.443885 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.443840 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn"] Nov 02 15:20:57.476838 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.476796 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.477063 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.476857 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.477151 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.477060 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.477151 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.477106 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.477269 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.477254 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xgvz\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-kube-api-access-6xgvz\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.477327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.477296 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.477396 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.477373 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.477519 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.477499 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.477609 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.477596 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579171 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579284 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579314 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579384 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579517 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579411 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579517 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579511 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xgvz\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-kube-api-access-6xgvz\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579687 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579661 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579827 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579657 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579827 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579742 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579827 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579758 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.579827 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579794 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.580054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.579839 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.580150 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.580128 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.580351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.580324 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.581952 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.581927 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.582291 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.582271 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.643646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.643619 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.643839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.643817 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xgvz\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-kube-api-access-6xgvz\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:57.656808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:57.656779 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:58.081810 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:58.081718 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn"] Nov 02 15:20:58.113343 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:20:58.113313 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cb52ef_3c0a_4efc_8879_f096233b6eab.slice/crio-a179241837dafcbd722a7ad2ec99fc2e2636286bf427de0053200e39af5ddf8e WatchSource:0}: Error finding container a179241837dafcbd722a7ad2ec99fc2e2636286bf427de0053200e39af5ddf8e: Status 404 returned error can't find the container with id a179241837dafcbd722a7ad2ec99fc2e2636286bf427de0053200e39af5ddf8e Nov 02 15:20:58.115927 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:58.115894 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.116062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:58.115965 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.116062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:58.116021 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.135547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:58.135519 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" event={"ID":"b8cb52ef-3c0a-4efc-8879-f096233b6eab","Type":"ContainerStarted","Data":"a179241837dafcbd722a7ad2ec99fc2e2636286bf427de0053200e39af5ddf8e"} Nov 02 15:20:59.102107 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:59.102067 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:20:59.249850 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:59.249568 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" event={"ID":"b8cb52ef-3c0a-4efc-8879-f096233b6eab","Type":"ContainerStarted","Data":"b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7"} Nov 02 15:20:59.413653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:59.413597 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" podStartSLOduration=2.4135843169999998 podStartE2EDuration="2.413584317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:59.410402791 +0000 UTC m=+1363.083606008" watchObservedRunningTime="2026-11-02 15:20:59.413584317 +0000 UTC m=+1363.086787511" Nov 02 15:20:59.656965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:59.656929 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:20:59.658456 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:59.658430 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:20:59.658567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:20:59.658483 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 15:21:00.657191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:00.657153 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:21:00.657665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:00.657228 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 15:21:01.657544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:01.657504 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:21:01.658081 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:01.657601 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 15:21:02.657585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:02.657550 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:21:02.658125 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:02.657613 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 15:21:03.658121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:03.658082 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:21:03.658548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:03.658139 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 15:21:04.079417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:04.079320 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:04.403600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:04.403567 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:04.657214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:04.657132 5433 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:21:04.657214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:04.657204 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 15:21:05.662048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:05.662013 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:21:05.662448 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:05.662421 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:21:05.663268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:05.663237 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:21:09.982001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:09.973081 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.015001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:10.014129 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:10.049000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:10.038057 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.099145 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:10.099095 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.100645 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:10.100608 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.447148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:10.447099 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.680322 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:10.680280 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.720057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:10.717068 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.037051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.029097 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:11.052013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.048095 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.052013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.048484 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.052013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.049129 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:21:11.052013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.049183 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:21:11.052013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.051278 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:21:11.209039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.202486 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.209039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.202605 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:21:11.399135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.388073 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:11.399135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.388144 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.575858 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.573586 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:11.575858 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.574770 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:21:11.593230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.584470 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:21:11.704066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.695883 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.736032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.735998 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:21:11.793998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.722217 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.922773 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.922740 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:21:11.958452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:11.958367 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:21:12.049059 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:12.048265 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:12.061643 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:12.054035 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:12.061643 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:12.054333 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:12.331062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:12.290881 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:21:12.360998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:12.291922 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:12.361634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:12.340132 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:21:12.497928 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:21:12.497885 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:21:12.607179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:12.607063 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:12.653428 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:12.653396 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:21:13.059938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:13.059375 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:13.086439 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:13.061091 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:13.297945 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:13.297900 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:21:13.372861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:13.372779 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:21:13.373286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:13.373270 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:21:13.374327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:13.372968 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:13.387454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:13.380852 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:21:17.226137 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:17.226100 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:22.258745 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:22.258707 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:23.466416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:23.466377 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:27.251147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.251105 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:27.291463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.291427 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm"] Nov 02 15:21:27.297435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.297412 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.405998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.396578 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cca276-7d0e-4481-a1d6-d6931fdac725-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.405998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.396639 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.405998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.396684 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.405998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.396732 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.405998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.396770 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.405998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.396924 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.405998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.397040 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.405998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.397105 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.405998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.397139 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcrz\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-kube-api-access-dkcrz\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.424711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.424681 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm"] Nov 02 15:21:27.498230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498194 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.498402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498239 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcrz\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-kube-api-access-dkcrz\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.498402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498321 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cca276-7d0e-4481-a1d6-d6931fdac725-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.498402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498351 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.498402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498390 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.498615 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498437 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.498615 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498476 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.498720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498640 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.498781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498725 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.498835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.498807 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.499114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.499084 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.499228 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.499096 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.499228 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.499122 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cca276-7d0e-4481-a1d6-d6931fdac725-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.499334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.499299 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.500688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.500665 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.501203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.501150 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.558071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.558030 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.558252 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.558133 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcrz\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-kube-api-access-dkcrz\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:27.661031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:27.660966 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:28.085372 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.085340 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm"] Nov 02 15:21:28.085956 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:21:28.085932 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cca276_7d0e_4481_a1d6_d6931fdac725.slice/crio-8e59928cfefd003ba160c65dddfe8c985eb1fd63bd6ab8662aef41cdc871659b WatchSource:0}: Error finding container 8e59928cfefd003ba160c65dddfe8c985eb1fd63bd6ab8662aef41cdc871659b: Status 404 returned error can't find the container with id 8e59928cfefd003ba160c65dddfe8c985eb1fd63bd6ab8662aef41cdc871659b Nov 02 15:21:28.088528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.088497 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:28.088622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.088561 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:28.088661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.088617 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:28.760605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.760564 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" event={"ID":"86cca276-7d0e-4481-a1d6-d6931fdac725","Type":"ContainerStarted","Data":"2fa2d3898c42c9114ebd820f1e256c3032b56bdc3cb28776e98f96e05fea0d32"} Nov 02 15:21:28.761051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.760623 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" event={"ID":"86cca276-7d0e-4481-a1d6-d6931fdac725","Type":"ContainerStarted","Data":"8e59928cfefd003ba160c65dddfe8c985eb1fd63bd6ab8662aef41cdc871659b"} Nov 02 15:21:28.788890 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.788856 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts"] Nov 02 15:21:28.793554 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.793531 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:28.888228 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.888191 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts"] Nov 02 15:21:28.943738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.932578 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podStartSLOduration=1.932558057 podStartE2EDuration="1.932558057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:28.931469236 +0000 UTC m=+1392.604672443" watchObservedRunningTime="2026-11-02 15:21:28.932558057 +0000 UTC m=+1392.605761254" Nov 02 15:21:28.946275 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.946240 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16240c61-804f-47e9-bfa4-57d049a4e508-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:28.947714 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.947678 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2z7b\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-kube-api-access-z2z7b\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:28.947852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.947739 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:28.947852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.947796 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:28.947958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.947878 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16240c61-804f-47e9-bfa4-57d049a4e508-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:28.948063 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.947962 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:28.948607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.948346 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:28.948812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.948774 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:28.948905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:28.948881 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050361 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050266 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050539 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050371 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050539 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050408 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050539 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050462 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050557 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16240c61-804f-47e9-bfa4-57d049a4e508-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050604 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2z7b\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-kube-api-access-z2z7b\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050638 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050679 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050758 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16240c61-804f-47e9-bfa4-57d049a4e508-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.050959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.050924 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.051188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.051162 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.051263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.051191 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.051306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.051270 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16240c61-804f-47e9-bfa4-57d049a4e508-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.051426 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.051407 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.052816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.052796 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.053238 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.053216 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16240c61-804f-47e9-bfa4-57d049a4e508-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.114630 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.114602 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.142646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.140798 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2z7b\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-kube-api-access-z2z7b\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.166164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.166135 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp"] Nov 02 15:21:29.171467 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.171451 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.255568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.255519 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp"] Nov 02 15:21:29.358053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.357946 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2ecad5-375c-461a-8f81-d2243d61c915-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.358203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.358107 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.358203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.358152 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.358268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.358236 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.358305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.358282 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.358354 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.358329 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.358417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.358405 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2hcc\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-kube-api-access-d2hcc\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.358453 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.358432 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.358484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.358468 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.408502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.408471 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:29.460014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.459962 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460082 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460123 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460224 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2hcc\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-kube-api-access-d2hcc\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460265 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460324 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460422 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460403 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2ecad5-375c-461a-8f81-d2243d61c915-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460507 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460487 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460506 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460520 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460542 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460819 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460798 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.460934 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.460911 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.461368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.461343 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2ecad5-375c-461a-8f81-d2243d61c915-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.462374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.462355 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.462746 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.462726 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.551066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.547878 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.551066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.550843 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2hcc\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-kube-api-access-d2hcc\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.587648 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.587446 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc"] Nov 02 15:21:29.594181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.594152 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.646998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.645142 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh"] Nov 02 15:21:29.653800 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.653192 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.653985 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.653833 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc"] Nov 02 15:21:29.666964 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.662211 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:29.666964 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.663865 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:21:29.666964 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.663913 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 15:21:29.708177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.708144 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh"] Nov 02 15:21:29.765955 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.765925 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknx8\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-kube-api-access-hknx8\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.766397 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766378 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.766557 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766543 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.766685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766662 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.766725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766698 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.766763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766726 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.766763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766757 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.766826 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766784 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.766826 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766802 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.766886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766833 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.766886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766849 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84888\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-kube-api-access-84888\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.766886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766867 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.767018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766895 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.767018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766913 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.767018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.766936 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.767129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.767057 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.767129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.767079 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.767129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.767104 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.786240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.786216 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:29.830159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.825639 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts"] Nov 02 15:21:29.857352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.855254 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.857352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.855352 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.857352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.855400 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.869284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869260 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.869395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869311 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.869395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869335 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.869395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869359 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.869395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869390 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.869601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869421 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.869601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869437 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.869601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869472 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.869601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869490 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84888\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-kube-api-access-84888\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.869601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869524 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.869601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869556 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.869601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869577 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.869924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869603 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.869924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869688 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.869924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869710 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.869924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869733 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.869924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869795 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknx8\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-kube-api-access-hknx8\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.869924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.869816 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.871817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.871795 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.874586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.873756 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.874697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.874469 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.875139 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.874887 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.875139 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.875102 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.875139 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.875074 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.875353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.875181 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.875447 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.875423 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.875570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.875550 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.875712 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.875693 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.875937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.875920 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.876099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.876077 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.877316 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.877293 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.879195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.879111 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.951262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.951214 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.953873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.953790 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84888\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-kube-api-access-84888\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.954022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.953868 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknx8\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-kube-api-access-hknx8\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:29.956244 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.956217 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:29.979828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:29.979795 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:30.202151 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.201946 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp"] Nov 02 15:21:30.204700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.204612 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.204816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.204750 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.204883 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.204822 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.215055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.215028 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:30.467005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.465931 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.467005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.466051 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.467005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.466108 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.479586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.479560 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh"] Nov 02 15:21:30.671437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.671399 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:21:30.671634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.671464 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 15:21:30.701905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.701813 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc"] Nov 02 15:21:30.808604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.806106 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.808604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.806194 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.808604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.806237 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.813420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.813194 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" event={"ID":"0b2ecad5-375c-461a-8f81-d2243d61c915","Type":"ContainerStarted","Data":"bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4"} Nov 02 15:21:30.813420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.813257 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" event={"ID":"0b2ecad5-375c-461a-8f81-d2243d61c915","Type":"ContainerStarted","Data":"b8e5b2f9c9c6191b9da50e80e4eebb3a7b2b2c7352daea3212dd1e34876dfa04"} Nov 02 15:21:30.815857 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.815598 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" event={"ID":"16240c61-804f-47e9-bfa4-57d049a4e508","Type":"ContainerStarted","Data":"e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0"} Nov 02 15:21:30.815857 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.815655 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" event={"ID":"16240c61-804f-47e9-bfa4-57d049a4e508","Type":"ContainerStarted","Data":"ea120edc1db00ed6637d570eded5c71c7a35c2bed530bc157d141d9310e8aefc"} Nov 02 15:21:30.836157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.818060 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" event={"ID":"d6f141fd-aa62-4bb8-90db-c5ee9518b194","Type":"ContainerStarted","Data":"55e72edde7cfc1ca42348519597874fa1a884ba98ab2c97af7cf53ae0b8ebba0"} Nov 02 15:21:30.836157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.825379 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" event={"ID":"f593f01e-a1ad-4ebd-b605-af6e203a8b7e","Type":"ContainerStarted","Data":"054d4d6a9dbe8bd7b7e9a5cbd17da346e3a0b9b7316357d0e4d674b996aa7eaa"} Nov 02 15:21:30.836157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.825426 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" event={"ID":"f593f01e-a1ad-4ebd-b605-af6e203a8b7e","Type":"ContainerStarted","Data":"d690a9227c5d74e532853f3167a2021188a30e03b263b0d86ba5ca1653491b2b"} Nov 02 15:21:30.982051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:30.980203 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:31.091000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.070592 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podStartSLOduration=2.070570384 podStartE2EDuration="2.070570384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:30.946631145 +0000 UTC m=+1394.619834342" watchObservedRunningTime="2026-11-02 15:21:31.070570384 +0000 UTC m=+1394.743773579" Nov 02 15:21:31.094404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.094344 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podStartSLOduration=2.094328919 podStartE2EDuration="2.094328919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:31.068907713 +0000 UTC m=+1394.742110910" watchObservedRunningTime="2026-11-02 15:21:31.094328919 +0000 UTC m=+1394.767532118" Nov 02 15:21:31.410000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.409349 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:31.429006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.428927 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.429205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.429025 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 15:21:31.664462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.664363 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.664718 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.664684 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 15:21:31.800005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.799089 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:31.800005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.799248 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.800005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.799297 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:31.899062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.884477 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" event={"ID":"d6f141fd-aa62-4bb8-90db-c5ee9518b194","Type":"ContainerStarted","Data":"db34fca964b0aac202e2f6ccb8dfc5a5c2c018e00dac826fff17cd951fd66b17"} Nov 02 15:21:32.000078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.992638 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.000078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:31.992692 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:32.035004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.034269 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podStartSLOduration=3.034248783 podStartE2EDuration="3.034248783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:32.033067524 +0000 UTC m=+1395.706270720" watchObservedRunningTime="2026-11-02 15:21:32.034248783 +0000 UTC m=+1395.707451979" Nov 02 15:21:32.053000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.050954 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podStartSLOduration=4.050925665 podStartE2EDuration="4.050925665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:31.203367076 +0000 UTC m=+1394.876570271" watchObservedRunningTime="2026-11-02 15:21:32.050925665 +0000 UTC m=+1395.724128861" Nov 02 15:21:32.228000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.226776 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:32.232995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.230073 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.232995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.230134 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:32.325058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.324069 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:21:32.404000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.396063 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:21:32.443000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.442149 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.443000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.442223 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 15:21:32.664000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.663390 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.664000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.663524 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 15:21:32.835002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.831880 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.835002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:32.831959 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:33.027097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.008128 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.106507 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.106436 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.106779 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.106750 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:33.130886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.110337 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:33.217948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.215406 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:33.217948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.216301 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.236017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.235947 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:33.297083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.296127 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:21:33.343412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.253121 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:33.343412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.333054 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:21:33.470175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.470135 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:21:33.503001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.473383 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:21:33.516552 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.516513 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.516823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.516787 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 15:21:33.596058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.536172 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:21:33.596858 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.596822 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:21:33.816010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.811116 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.901115 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.901069 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:33.921086 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.817090 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.921425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:33.921384 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 15:21:34.007715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.007676 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.008126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.008075 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:34.008416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.008390 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.010134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.010109 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:21:34.216115 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.216018 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.239156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.239113 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:21:34.290240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.281710 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.290240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.281771 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:34.301124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.301084 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:21:34.302719 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.302695 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:21:34.336906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.336862 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.453002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.452165 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.453002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.452228 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 15:21:34.610183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.610070 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.666062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.666029 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.666320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.666290 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 15:21:34.666490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.666470 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.795583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.795548 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.795850 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.795822 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:34.901083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:34.875258 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:21:35.053000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.031544 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.110326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.079739 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.134336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.100428 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.134621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.134571 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:35.257052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.224242 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.283562 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.283527 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.475003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.409605 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:35.475003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.367852 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.475003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.365681 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.475003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.411191 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.475003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.411228 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 15:21:35.619319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.619212 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.688010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.675112 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.795554 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.795519 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.795813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.795788 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:35.796205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.796182 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.796372 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.796351 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 15:21:35.992035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.991134 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.992035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:35.991195 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:36.126000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.119063 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.144005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.138291 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.144005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.138961 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:21:36.251180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.246804 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.251180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.246863 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:36.352673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.267355 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.458674 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.458626 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:21:36.459390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.459344 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:21:36.619198 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.619097 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.619476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.619444 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 15:21:36.776004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.775941 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:21:36.951362 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.776271 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.019107 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.845836 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.019408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.019374 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:37.019551 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.980329 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.019674 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.019655 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:37.019811 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:36.956195 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Nov 02 15:21:37.127283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.119273 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 15:21:37.147513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.147464 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.235355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.235262 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.235617 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.235587 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:37.418019 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.415741 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:37.418019 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.416852 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.418019 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.416897 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 15:21:37.485116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.485063 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.519061 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.486192 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:21:37.573179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.573112 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.791628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.791535 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.792398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.792367 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:37.816230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.816187 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.915110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.915071 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:37.924999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.915359 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.961743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:37.961689 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.020026 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.004376 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.020026 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.004439 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:38.022112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.021666 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.132286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.132246 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.186280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.185950 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:38.247005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.246310 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:38.279636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.247922 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:38.279636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.250932 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.279636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.250997 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:38.285191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.285155 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:21:38.439620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.425513 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.505641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.486066 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.523479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.523435 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.524941 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.524912 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.543230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.543192 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.581151 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.581104 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 15:21:38.729056 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.678071 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.812006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.793047 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.812006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.793108 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:38.969117 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:38.969066 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.057086 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.049055 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.132999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.132304 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:39.132999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.132381 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:39.132999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.132897 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 15:21:39.200121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.200068 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.269736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.247492 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:39.269736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.247552 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:39.303017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.293099 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.335747 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.335662 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:21:39.414962 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.414923 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:21:39.415249 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.415211 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 15:21:39.435957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.435080 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:39.502117 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.491115 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.527899 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.527853 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.796107 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.796058 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:39.796301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:39.796124 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:40.041626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.041585 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:21:40.204164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.143088 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:40.204164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.143150 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:40.204164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.179144 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:40.244999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.228813 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:21:40.257148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.257112 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:40.272252 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.272204 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:40.277572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.273293 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:40.277572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.273347 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:40.489230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.486318 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:21:40.605999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.563670 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:21:40.650724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.650668 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:21:40.651032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.574196 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:40.693933 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.693899 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:21:40.701719 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.701693 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:21:40.791290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.791181 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 15:21:40.791606 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.791574 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 15:21:40.989915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.983412 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:21:40.989915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:40.983475 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 15:21:41.083383 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:21:41.083280 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:21:41.098932 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:41.095998 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:41.098932 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:41.098626 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:41.101613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:41.101544 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:21:41.216395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:41.216357 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:21:41.216585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:41.216412 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 15:21:42.117737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:42.117707 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:42.118853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:42.118827 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:42.124078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:42.124055 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:21:42.188238 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:42.188191 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:42.189159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:42.189120 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:42.190065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:42.190046 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:21:42.225532 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:42.225506 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:42.226135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:42.226106 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:42.227365 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:42.227343 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:21:44.395646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.393923 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv"] Nov 02 15:21:44.416409 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.416381 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.445473 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.445436 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv"] Nov 02 15:21:44.513135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.513098 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.513418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.513384 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.513558 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.513503 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.513624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.513573 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b4a4e-eb71-4780-8590-79c2a11ab95a-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.513791 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.513746 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxt8\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-kube-api-access-chxt8\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.513866 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.513809 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.513866 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.513844 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.514042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.514017 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.514111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.514081 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.622016 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.622083 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.622172 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.622283 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b4a4e-eb71-4780-8590-79c2a11ab95a-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.622321 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxt8\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-kube-api-access-chxt8\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.622383 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.622417 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.622442 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.622498 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.662069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.633304 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.720033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.664885 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.731390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.721438 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.742992 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.742917 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b4a4e-eb71-4780-8590-79c2a11ab95a-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.874307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.874191 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.875649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.875587 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxt8\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-kube-api-access-chxt8\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.876960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.876224 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.877434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.877412 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:44.928730 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:44.928588 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:45.048524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:45.048359 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:46.050007 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:46.049101 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:21:46.154000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:46.149143 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.192953 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:46.183574 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.485140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:46.485081 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.524011 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:46.516354 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.524422 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:46.524267 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.753067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:46.674923 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:21:46.753067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:46.737428 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:46.819340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:46.819217 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.148318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.148272 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.164289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.164259 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:21:47.195143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.195095 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.225519 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.164541 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.230452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.230059 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:21:47.238999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.230698 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:21:47.684620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.664903 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.684620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.665291 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:47.684620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.671724 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Nov 02 15:21:47.684620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.671798 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:21:47.684620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.680567 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:21:47.684620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.680637 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:21:47.684620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.680677 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:21:47.733080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.718170 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.733884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.733859 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:21:47.814963 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:47.814910 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.196521 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.196473 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.240476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.240369 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:48.240762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.240734 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:48.240923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.240902 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:48.288541 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.288485 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.298025 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.293844 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.648065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.648006 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Nov 02 15:21:48.694012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.667125 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:48.773726 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.675014 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:21:48.787002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.675091 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.805036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.682116 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 02 15:21:48.863831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.734309 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.901104 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.901013 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:48.904107 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:48.904063 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.235058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.209550 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.289923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.289881 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 02 15:21:49.310678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.310628 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.311725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.311690 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 15:21:49.777415 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.776678 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:21:49.777415 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.777100 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:49.797736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.797682 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.833774 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:21:49.830116 5433 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.378s" Nov 02 15:21:49.833774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.830256 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv"] Nov 02 15:21:49.870001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.866430 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.937436 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:49.937318 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" event={"ID":"426b4a4e-eb71-4780-8590-79c2a11ab95a","Type":"ContainerStarted","Data":"d7e962490b9739c94195c802a4c497341d067f54c0e939726e5c96bdd528a6bb"} Nov 02 15:21:50.118124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.118082 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:50.118401 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.118366 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.243211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.243103 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.329999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.305332 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.330287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.313241 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:50.337339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.325360 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.657077 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.657041 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:21:50.886137 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.847013 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:21:50.886757 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.847288 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:50.908071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.893070 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:21:50.908071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:50.907201 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:51.477062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:51.375414 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:21:51.662054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:51.659425 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:21:51.915415 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:51.894090 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:21:52.228053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:51.987890 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:52.229138 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:52.045475 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:21:52.229930 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:52.045515 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:52.241455 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:52.241405 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" event={"ID":"426b4a4e-eb71-4780-8590-79c2a11ab95a","Type":"ContainerStarted","Data":"3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3"} Nov 02 15:21:52.424896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:52.387700 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:21:52.763587 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:52.761829 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:52.919219 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:52.919167 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:53.082117 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.082023 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:21:53.082357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.082322 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 15:21:53.262001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.231999 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:53.262001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.232347 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:21:53.262001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.239027 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:21:53.274569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.274500 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" podStartSLOduration=9.274482056 podStartE2EDuration="9.274482056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:52.954317012 +0000 UTC m=+1416.627520205" watchObservedRunningTime="2026-11-02 15:21:53.274482056 +0000 UTC m=+1416.947685252" Nov 02 15:21:53.464030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.435771 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:53.500000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.492746 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:21:53.711777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.708563 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:21:53.743650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.743562 5433 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:53.743895 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.743865 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:53.793533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.765140 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:53.793533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.765468 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:21:53.903013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.894652 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2"] Nov 02 15:21:53.903013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.894894 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" containerID="cri-o://22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc" gracePeriod=30 Nov 02 15:21:53.940167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.940126 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:53.966029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:53.964116 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:21:54.054215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:54.052536 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:21:54.054215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:54.052603 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 15:21:54.055592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:54.055506 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:21:54.097233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:54.097202 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:21:54.189356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:54.189323 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:21:54.194821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:54.194796 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:21:54.205992 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:54.202624 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:21:55.068008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:55.055314 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:21:55.068008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:55.055376 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 15:21:55.548169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:55.548128 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:21:56.049938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:56.049900 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:21:56.050122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:56.049966 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 15:21:57.049961 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:57.049924 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:21:57.050402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:57.049999 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 15:21:58.049425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:58.049387 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:21:58.049609 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:58.049451 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 15:21:59.049695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.049659 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:21:59.050095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.049727 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 15:21:59.201648 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.201621 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:21:59.351547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.351516 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9da2d9-84e4-4805-863f-3b848377703d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9da2d9-84e4-4805-863f-3b848377703d-istiod-ca-cert\") pod \"da9da2d9-84e4-4805-863f-3b848377703d\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " Nov 02 15:21:59.351740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.351609 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-credential-socket\") pod \"da9da2d9-84e4-4805-863f-3b848377703d\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " Nov 02 15:21:59.351740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.351724 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-istio-token\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-istio-token\") pod \"da9da2d9-84e4-4805-863f-3b848377703d\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " Nov 02 15:21:59.351936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.351836 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-kube-api-access-lsngf\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-kube-api-access-lsngf\") pod \"da9da2d9-84e4-4805-863f-3b848377703d\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " Nov 02 15:21:59.351936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.351886 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-envoy\") pod \"da9da2d9-84e4-4805-863f-3b848377703d\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " Nov 02 15:21:59.352094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.351932 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-credential-socket" pod "da9da2d9-84e4-4805-863f-3b848377703d" (UID: "da9da2d9-84e4-4805-863f-3b848377703d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.352094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.352004 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da9da2d9-84e4-4805-863f-3b848377703d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9da2d9-84e4-4805-863f-3b848377703d-istio-podinfo\") pod \"da9da2d9-84e4-4805-863f-3b848377703d\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " Nov 02 15:21:59.352094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.352073 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-data\") pod \"da9da2d9-84e4-4805-863f-3b848377703d\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " Nov 02 15:21:59.352261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.352111 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-certs\") pod \"da9da2d9-84e4-4805-863f-3b848377703d\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " Nov 02 15:21:59.352261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.352124 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9da2d9-84e4-4805-863f-3b848377703d-istiod-ca-cert" pod "da9da2d9-84e4-4805-863f-3b848377703d" (UID: "da9da2d9-84e4-4805-863f-3b848377703d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:59.352261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.352162 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-socket\") pod \"da9da2d9-84e4-4805-863f-3b848377703d\" (UID: \"da9da2d9-84e4-4805-863f-3b848377703d\") " Nov 02 15:21:59.353410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.352779 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9da2d9-84e4-4805-863f-3b848377703d-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.353410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.352802 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.353410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.353053 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-socket" pod "da9da2d9-84e4-4805-863f-3b848377703d" (UID: "da9da2d9-84e4-4805-863f-3b848377703d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.353410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.353376 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-certs" pod "da9da2d9-84e4-4805-863f-3b848377703d" (UID: "da9da2d9-84e4-4805-863f-3b848377703d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.353948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.353925 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-data" pod "da9da2d9-84e4-4805-863f-3b848377703d" (UID: "da9da2d9-84e4-4805-863f-3b848377703d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.356634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.356610 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-istio-token" pod "da9da2d9-84e4-4805-863f-3b848377703d" (UID: "da9da2d9-84e4-4805-863f-3b848377703d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:59.358141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.358095 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da9da2d9-84e4-4805-863f-3b848377703d-istio-podinfo" pod "da9da2d9-84e4-4805-863f-3b848377703d" (UID: "da9da2d9-84e4-4805-863f-3b848377703d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:21:59.358438 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.358410 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-kube-api-access-lsngf" pod "da9da2d9-84e4-4805-863f-3b848377703d" (UID: "da9da2d9-84e4-4805-863f-3b848377703d"). InnerVolumeSpecName "kube-api-access-lsngf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:59.358542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.358447 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-envoy" pod "da9da2d9-84e4-4805-863f-3b848377703d" (UID: "da9da2d9-84e4-4805-863f-3b848377703d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.403002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.402956 5433 generic.go:356] "Generic (PLEG): container finished" podID="da9da2d9-84e4-4805-863f-3b848377703d" containerID="22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc" exitCode=0 Nov 02 15:21:59.403162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.403015 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" Nov 02 15:21:59.403162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.403024 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" event={"ID":"da9da2d9-84e4-4805-863f-3b848377703d","Type":"ContainerDied","Data":"22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc"} Nov 02 15:21:59.403162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.403097 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2" event={"ID":"da9da2d9-84e4-4805-863f-3b848377703d","Type":"ContainerDied","Data":"eb4a41e510a5afaf0518a028afe0d33cef95572d9c7498d4602a7f46ba7bb7e0"} Nov 02 15:21:59.403162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.403121 5433 scope.go:122] "RemoveContainer" containerID="22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc" Nov 02 15:21:59.415825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.415801 5433 scope.go:122] "RemoveContainer" containerID="22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc" Nov 02 15:21:59.416177 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:21:59.416154 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc\": container with ID starting with 22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc not found: ID does not exist" containerID="22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc" Nov 02 15:21:59.416267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.416187 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc"} err="failed to get container status \"22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc\": rpc error: code = NotFound desc = could not find container \"22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc\": container with ID starting with 22bcbbce4b1af8ad6c3d2e3d010077abc25f1b452f871609a1d420796eb1c7fc not found: ID does not exist" Nov 02 15:21:59.454122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.454045 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9da2d9-84e4-4805-863f-3b848377703d-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.454122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.454082 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.454122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.454097 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.454122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.454113 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.454122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.454128 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.454491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.454142 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsngf\" (UniqueName: \"kubernetes.io/projected/da9da2d9-84e4-4805-863f-3b848377703d-kube-api-access-lsngf\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.454491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.454158 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9da2d9-84e4-4805-863f-3b848377703d-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.559101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.559068 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2"] Nov 02 15:21:59.563496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:21:59.563468 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-f5wz2"] Nov 02 15:22:00.049163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:00.049128 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:22:00.087365 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:00.087121 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:22:00.411113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:00.411086 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:22:00.458567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:00.458516 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9da2d9-84e4-4805-863f-3b848377703d" path="/var/lib/kubelet/pods/da9da2d9-84e4-4805-863f-3b848377703d/volumes" Nov 02 15:22:00.728610 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:00.728521 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:05.773017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:05.772956 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:10.781902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:10.781868 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:11.011835 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:22:11.011805 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:22:11.927041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:11.926966 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:12.083302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:12.083265 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:12.753121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:12.753088 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:13.442736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:13.442701 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:14.413328 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.413296 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg"] Nov 02 15:22:14.414003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.413964 5433 cpu_manager.go:414] "containerMap: removing container" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" Nov 02 15:22:14.414003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.413992 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" Nov 02 15:22:14.414118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.414113 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9da2d9-84e4-4805-863f-3b848377703d" containerName="istio-proxy" Nov 02 15:22:14.418153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.418131 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.466963 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.466931 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a758f7bf-openshift-default-dockercfg-86zj5\"" Nov 02 15:22:14.502271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.502231 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:14.518342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.518313 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg"] Nov 02 15:22:14.519068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.519039 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-socket\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.519227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.519213 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-podinfo\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.519312 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.519267 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-token\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.519370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.519325 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-certs\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.519370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.519366 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-data\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.519476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.519388 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz7ch\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-kube-api-access-kz7ch\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.519476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.519425 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istiod-ca-cert\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.519604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.519586 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-envoy\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.519664 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.519622 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-credential-socket\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.629331 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-podinfo\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.629426 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-token\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.629489 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-certs\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.629549 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-data\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.629627 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7ch\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-kube-api-access-kz7ch\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.629687 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istiod-ca-cert\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.629777 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-envoy\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.629822 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-credential-socket\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.629900 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-socket\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.630996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.630583 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-socket\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.643823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.643138 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-certs\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.643823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.643483 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-data\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.643823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.643770 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-credential-socket\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.644771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.644329 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istiod-ca-cert\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.647723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.647697 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-podinfo\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.672051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.659320 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-envoy\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.740128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.726184 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:14.740128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.726961 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-token\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.740128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.728580 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7ch\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-kube-api-access-kz7ch\") pod \"e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:14.740428 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:14.740377 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:15.243299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:15.243266 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg"] Nov 02 15:22:15.245048 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:22:15.245015 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c40bb21_2237_4cb8_8792_9e3dd99d62f7.slice/crio-266ea17960853404865ddcbbcc0cf9a9667a213b3500429a06c5a6d6ae343801 WatchSource:0}: Error finding container 266ea17960853404865ddcbbcc0cf9a9667a213b3500429a06c5a6d6ae343801: Status 404 returned error can't find the container with id 266ea17960853404865ddcbbcc0cf9a9667a213b3500429a06c5a6d6ae343801 Nov 02 15:22:15.247665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:15.247630 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:15.247765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:15.247705 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:15.247765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:15.247747 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:15.513348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:15.513246 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" event={"ID":"4c40bb21-2237-4cb8-8792-9e3dd99d62f7","Type":"ContainerStarted","Data":"b057dd3dbacc95a3a0616ccda8ca6e3c7c0c6503486e6834ffeedaf45132e3cd"} Nov 02 15:22:15.513348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:15.513308 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" event={"ID":"4c40bb21-2237-4cb8-8792-9e3dd99d62f7","Type":"ContainerStarted","Data":"266ea17960853404865ddcbbcc0cf9a9667a213b3500429a06c5a6d6ae343801"} Nov 02 15:22:15.629616 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:15.629555 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" podStartSLOduration=1.629533423 podStartE2EDuration="1.629533423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:15.626765029 +0000 UTC m=+1439.299968225" watchObservedRunningTime="2026-11-02 15:22:15.629533423 +0000 UTC m=+1439.302736619" Nov 02 15:22:15.754266 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:15.752493 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:16.206081 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:16.206045 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:16.747246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:16.747207 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:17.528296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:17.528262 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:17.529165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:17.529144 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:22:17.813451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:17.813364 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:18.014600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:18.014565 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:19.965831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:19.965795 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:20.140082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:20.140044 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:20.293783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:20.293683 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:20.480780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:20.480751 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:20.641951 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:20.641915 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:20.807856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:20.807820 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:21.163998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.158311 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:21.337036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.336997 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:21.627154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.616782 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz"] Nov 02 15:22:21.692856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.692098 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:21.709998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.700366 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz"] Nov 02 15:22:21.749947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.749640 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrg54\" (UniqueName: \"kubernetes.io/projected/1d3469de-4872-4276-ab00-96856badbde3-kube-api-access-xrg54\") pod \"payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz\" (UID: \"1d3469de-4872-4276-ab00-96856badbde3\") " pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:21.756594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.756561 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3469de-4872-4276-ab00-96856badbde3-plugins-config-volume\") pod \"payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz\" (UID: \"1d3469de-4872-4276-ab00-96856badbde3\") " pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:21.757012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.756386 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a758f7bf\"" Nov 02 15:22:21.757166 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.756314 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a758f7bf-dockercfg-252kc\"" Nov 02 15:22:21.779814 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.776839 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:21.795585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.794064 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:21.889149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.889044 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrg54\" (UniqueName: \"kubernetes.io/projected/1d3469de-4872-4276-ab00-96856badbde3-kube-api-access-xrg54\") pod \"payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz\" (UID: \"1d3469de-4872-4276-ab00-96856badbde3\") " pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:21.929778 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.929739 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3469de-4872-4276-ab00-96856badbde3-plugins-config-volume\") pod \"payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz\" (UID: \"1d3469de-4872-4276-ab00-96856badbde3\") " pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:21.953765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.953731 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3469de-4872-4276-ab00-96856badbde3-plugins-config-volume\") pod \"payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz\" (UID: \"1d3469de-4872-4276-ab00-96856badbde3\") " pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:22.002013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:21.999620 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrg54\" (UniqueName: \"kubernetes.io/projected/1d3469de-4872-4276-ab00-96856badbde3-kube-api-access-xrg54\") pod \"payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz\" (UID: \"1d3469de-4872-4276-ab00-96856badbde3\") " pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:22.195281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:22.190305 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:22:22.246264 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:22.246218 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.282508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:22.246611 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:22:22.298340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:22.298301 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:22.347224 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:22.346466 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 15:22:22.792854 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:22.792741 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:22:22.991999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:22.802185 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.991999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:22.803428 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:22:22.991999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:22.971966 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:23.257348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.257306 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.282037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.258944 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:23.282037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.261793 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:22:23.282037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.278852 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:22:23.282037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.278894 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:22:23.291733 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.291696 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.299938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.299889 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:22:23.309114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.309080 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:22:23.309385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.309370 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:22:23.386999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.355436 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.387447 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.387416 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:22:23.664333 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.599528 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:23.988476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.988432 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.989105 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.989073 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.989526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.989505 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:22:24.026001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:23.994147 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:22:24.355110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:24.307866 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:22:25.451419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:24.307646 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.664862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:24.371070 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:22:25.665527 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:24.688809 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:25.665762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:24.638957 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 15:22:25.666046 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:24.999778 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.674075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:24.999840 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:22:25.674448 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:24.999878 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:22:25.674736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:25.414600 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz"] Nov 02 15:22:25.944014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:25.933660 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:22:25.944014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:25.933719 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.097152 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.088271 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" event={"ID":"1d3469de-4872-4276-ab00-96856badbde3","Type":"ContainerStarted","Data":"2c17ff1c3c50e30ae835be8ff02777e38d65181c0fc584e46ba098a33a0293dc"} Nov 02 15:22:26.324311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.324232 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:22:26.324572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.324528 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.425141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.425096 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.919846 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.919802 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:22:27.038649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.922127 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:22:27.064998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.922250 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 15:22:27.067489 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.922340 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:22:27.067734 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.922523 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:27.067865 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:26.922613 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:22:27.068252 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:27.068213 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Nov 02 15:22:27.441214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:27.441162 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:27.616904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:27.616861 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:22:27.617226 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:27.617178 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:27.635319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:27.635273 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:28.291639 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:22:28.291601 5433 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.669s" Nov 02 15:22:28.292476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.292447 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:22:28.338632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.338594 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:28.367986 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.363988 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" event={"ID":"1d3469de-4872-4276-ab00-96856badbde3","Type":"ContainerStarted","Data":"1b6ab45cc0f1503ce0a8e0b702059c91b1d1dac00abc2c59207fd10835ee9632"} Nov 02 15:22:28.367986 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.365057 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:28.415006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.414305 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:22:28.448942 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.448914 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:22:28.532911 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.526558 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:28.558825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.558735 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:22:28.559088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.559068 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:22:28.567939 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.567919 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:22:28.614172 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.614142 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:22:28.666882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:28.666821 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" podStartSLOduration=7.666800434 podStartE2EDuration="7.666800434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:28.663280075 +0000 UTC m=+1452.336483289" watchObservedRunningTime="2026-11-02 15:22:28.666800434 +0000 UTC m=+1452.340003629" Nov 02 15:22:30.122835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:30.122799 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:30.863358 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:30.863319 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:31.209964 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.209880 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:31.307482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.306401 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d"] Nov 02 15:22:31.413268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.413231 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d"] Nov 02 15:22:31.413620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.413603 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:31.476796 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.476757 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7"] Nov 02 15:22:31.498451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.498415 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bd9398b4-dockercfg-q445c\"" Nov 02 15:22:31.503867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.500570 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bd9398b4\"" Nov 02 15:22:31.523014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.510740 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:31.557522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.557473 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6c89\" (UniqueName: \"kubernetes.io/projected/ea639834-f676-4602-8ea3-265b074e2d10-kube-api-access-q6c89\") pod \"payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d\" (UID: \"ea639834-f676-4602-8ea3-265b074e2d10\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:31.558128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.558101 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea639834-f676-4602-8ea3-265b074e2d10-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d\" (UID: \"ea639834-f676-4602-8ea3-265b074e2d10\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:31.580915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.563696 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7"] Nov 02 15:22:31.692180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.692103 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6c89\" (UniqueName: \"kubernetes.io/projected/ea639834-f676-4602-8ea3-265b074e2d10-kube-api-access-q6c89\") pod \"payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d\" (UID: \"ea639834-f676-4602-8ea3-265b074e2d10\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:31.692491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.692469 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b025e5-4c21-4036-91b2-c97f853feee9-plugins-config-volume\") pod \"payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7\" (UID: \"81b025e5-4c21-4036-91b2-c97f853feee9\") " pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:31.692836 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.692818 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea639834-f676-4602-8ea3-265b074e2d10-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d\" (UID: \"ea639834-f676-4602-8ea3-265b074e2d10\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:31.692960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.692946 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfb4\" (UniqueName: \"kubernetes.io/projected/81b025e5-4c21-4036-91b2-c97f853feee9-kube-api-access-9pfb4\") pod \"payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7\" (UID: \"81b025e5-4c21-4036-91b2-c97f853feee9\") " pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:31.731933 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.725606 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea639834-f676-4602-8ea3-265b074e2d10-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d\" (UID: \"ea639834-f676-4602-8ea3-265b074e2d10\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:31.784028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.776691 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6c89\" (UniqueName: \"kubernetes.io/projected/ea639834-f676-4602-8ea3-265b074e2d10-kube-api-access-q6c89\") pod \"payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d\" (UID: \"ea639834-f676-4602-8ea3-265b074e2d10\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:31.823060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.812917 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:31.823060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.821998 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfb4\" (UniqueName: \"kubernetes.io/projected/81b025e5-4c21-4036-91b2-c97f853feee9-kube-api-access-9pfb4\") pod \"payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7\" (UID: \"81b025e5-4c21-4036-91b2-c97f853feee9\") " pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:31.823060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.822263 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b025e5-4c21-4036-91b2-c97f853feee9-plugins-config-volume\") pod \"payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7\" (UID: \"81b025e5-4c21-4036-91b2-c97f853feee9\") " pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:31.823833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.823286 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b025e5-4c21-4036-91b2-c97f853feee9-plugins-config-volume\") pod \"payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7\" (UID: \"81b025e5-4c21-4036-91b2-c97f853feee9\") " pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:31.883067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:31.883037 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfb4\" (UniqueName: \"kubernetes.io/projected/81b025e5-4c21-4036-91b2-c97f853feee9-kube-api-access-9pfb4\") pod \"payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7\" (UID: \"81b025e5-4c21-4036-91b2-c97f853feee9\") " pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:32.056451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:32.056365 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:32.194710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:32.194677 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d"] Nov 02 15:22:32.196036 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:22:32.196006 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea639834_f676_4602_8ea3_265b074e2d10.slice/crio-ede9db9b63f9df509f2cb0b3ebaac88f51c0b8b78232ed18835e96325eb9fd1e WatchSource:0}: Error finding container ede9db9b63f9df509f2cb0b3ebaac88f51c0b8b78232ed18835e96325eb9fd1e: Status 404 returned error can't find the container with id ede9db9b63f9df509f2cb0b3ebaac88f51c0b8b78232ed18835e96325eb9fd1e Nov 02 15:22:32.431871 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:32.431845 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7"] Nov 02 15:22:32.434175 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:22:32.434147 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b025e5_4c21_4036_91b2_c97f853feee9.slice/crio-ac07daf977ea82c248a55bea7ded2a1f4a3e01a649f4afbb9fcdde94b6821f8b WatchSource:0}: Error finding container ac07daf977ea82c248a55bea7ded2a1f4a3e01a649f4afbb9fcdde94b6821f8b: Status 404 returned error can't find the container with id ac07daf977ea82c248a55bea7ded2a1f4a3e01a649f4afbb9fcdde94b6821f8b Nov 02 15:22:32.469791 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:32.469764 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" event={"ID":"ea639834-f676-4602-8ea3-265b074e2d10","Type":"ContainerStarted","Data":"82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3"} Nov 02 15:22:32.469915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:32.469812 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" event={"ID":"ea639834-f676-4602-8ea3-265b074e2d10","Type":"ContainerStarted","Data":"ede9db9b63f9df509f2cb0b3ebaac88f51c0b8b78232ed18835e96325eb9fd1e"} Nov 02 15:22:32.469915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:32.469859 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:32.471218 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:32.471192 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" event={"ID":"81b025e5-4c21-4036-91b2-c97f853feee9","Type":"ContainerStarted","Data":"ac07daf977ea82c248a55bea7ded2a1f4a3e01a649f4afbb9fcdde94b6821f8b"} Nov 02 15:22:32.581308 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:32.581253 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" podStartSLOduration=1.581239131 podStartE2EDuration="1.581239131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:32.578388757 +0000 UTC m=+1456.251591951" watchObservedRunningTime="2026-11-02 15:22:32.581239131 +0000 UTC m=+1456.254442326" Nov 02 15:22:33.061808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:33.061770 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:33.273160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:33.273120 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:33.479560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:33.479523 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" event={"ID":"81b025e5-4c21-4036-91b2-c97f853feee9","Type":"ContainerStarted","Data":"7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184"} Nov 02 15:22:33.589672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:33.589614 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" podStartSLOduration=2.58959628 podStartE2EDuration="2.58959628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:33.587997581 +0000 UTC m=+1457.261200780" watchObservedRunningTime="2026-11-02 15:22:33.58959628 +0000 UTC m=+1457.262799476" Nov 02 15:22:34.492083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:34.492050 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:35.426966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:35.426925 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:35.576613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:35.576579 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:35.732554 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:35.732467 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:35.915037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:35.910943 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:36.095062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.094909 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:36.394321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.394270 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns"] Nov 02 15:22:36.808683 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.807847 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:36.837500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.837465 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svsj\" (UniqueName: \"kubernetes.io/projected/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-kube-api-access-7svsj\") pod \"payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns\" (UID: \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\") " pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:36.837799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.837771 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-plugins-config-volume\") pod \"payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns\" (UID: \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\") " pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:36.857172 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.857139 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns"] Nov 02 15:22:36.888020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.887951 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-06a527d4\"" Nov 02 15:22:36.934044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.934017 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-06a527d4-dockercfg-lvm4t\"" Nov 02 15:22:36.939315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.939288 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svsj\" (UniqueName: \"kubernetes.io/projected/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-kube-api-access-7svsj\") pod \"payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns\" (UID: \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\") " pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:36.939405 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.939330 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-plugins-config-volume\") pod \"payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns\" (UID: \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\") " pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:36.940050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.940006 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-plugins-config-volume\") pod \"payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns\" (UID: \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\") " pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:36.996599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:36.996569 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svsj\" (UniqueName: \"kubernetes.io/projected/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-kube-api-access-7svsj\") pod \"payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns\" (UID: \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\") " pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:37.163801 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:37.163771 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:37.535473 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:37.535446 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns"] Nov 02 15:22:37.537375 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:22:37.537347 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda262cd1f_20d5_40ee_82cc_d669bcbf8b42.slice/crio-0cab960e18dd3762bb6700b6e6eb54aae283063c75e156c577b43961522f2c60 WatchSource:0}: Error finding container 0cab960e18dd3762bb6700b6e6eb54aae283063c75e156c577b43961522f2c60: Status 404 returned error can't find the container with id 0cab960e18dd3762bb6700b6e6eb54aae283063c75e156c577b43961522f2c60 Nov 02 15:22:37.616707 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:37.616556 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" event={"ID":"a262cd1f-20d5-40ee-82cc-d669bcbf8b42","Type":"ContainerStarted","Data":"0cab960e18dd3762bb6700b6e6eb54aae283063c75e156c577b43961522f2c60"} Nov 02 15:22:38.624710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:38.624672 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" event={"ID":"a262cd1f-20d5-40ee-82cc-d669bcbf8b42","Type":"ContainerStarted","Data":"6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa"} Nov 02 15:22:38.627110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:38.624738 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:38.739625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:38.739565 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" podStartSLOduration=2.739546143 podStartE2EDuration="2.739546143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:38.736701883 +0000 UTC m=+1462.409905078" watchObservedRunningTime="2026-11-02 15:22:38.739546143 +0000 UTC m=+1462.412749339" Nov 02 15:22:40.406571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:40.406541 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:22:41.015089 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:22:41.015052 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:22:41.449844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:41.449808 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:43.483252 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:43.483211 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:22:44.462378 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.462340 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc"] Nov 02 15:22:44.466748 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.466724 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.508408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.508375 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf"] Nov 02 15:22:44.512496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.512471 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630072 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630147 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630190 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630224 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630249 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4r7n\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-kube-api-access-q4r7n\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630311 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630335 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.630398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630363 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.630490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630399 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.630490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630471 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.630551 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630495 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4wpf\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-kube-api-access-c4wpf\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.630586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630569 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d224219-f041-4188-a745-e0bf2e46f1b4-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630643 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630626 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630658 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.630730 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630694 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.630768 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630731 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.630768 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630759 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.630838 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.630814 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.661499 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.661467 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc"] Nov 02 15:22:44.732469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732392 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.732469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732431 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.732469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732455 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.732754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732493 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.732754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732559 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.732754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732586 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4wpf\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-kube-api-access-c4wpf\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.732754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732654 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d224219-f041-4188-a745-e0bf2e46f1b4-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.732754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732716 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.732754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732742 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.733088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732779 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.733088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732815 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.733088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732846 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.733088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732847 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.733088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732910 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.733088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.732915 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.733088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733029 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.733461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733143 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.733461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733203 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.733461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733214 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.733461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733244 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.733461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733279 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.733461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733305 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r7n\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-kube-api-access-q4r7n\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.733461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733416 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.733816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733695 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.733816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733795 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.733933 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.733830 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d224219-f041-4188-a745-e0bf2e46f1b4-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.734035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.734000 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.734108 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.734093 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.735055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.735029 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.735887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.735862 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.736008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.735868 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.736008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.735948 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.788770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.788749 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf"] Nov 02 15:22:44.794386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.794338 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.794598 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.794577 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.794671 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.794629 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4r7n\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-kube-api-access-q4r7n\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:44.794738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.794715 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4wpf\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-kube-api-access-c4wpf\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:44.826653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:44.826625 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:45.081231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.081145 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:45.256797 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.256765 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf"] Nov 02 15:22:45.257636 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:22:45.257608 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d224219_f041_4188_a745_e0bf2e46f1b4.slice/crio-aafd28db9bb587e605c8cdbdacafda671c8b02a841f77c7cabde56f27184ab52 WatchSource:0}: Error finding container aafd28db9bb587e605c8cdbdacafda671c8b02a841f77c7cabde56f27184ab52: Status 404 returned error can't find the container with id aafd28db9bb587e605c8cdbdacafda671c8b02a841f77c7cabde56f27184ab52 Nov 02 15:22:45.260454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.260422 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.260536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.260489 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.260536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.260518 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.460326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.460294 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc"] Nov 02 15:22:45.461590 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:22:45.461563 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf4f2d7_b4ae_476c_b924_7f2ad9beef00.slice/crio-786e2fc082e8583771ea01ea87ff2782d2d2b073ff900c2185d66db3d5a906b9 WatchSource:0}: Error finding container 786e2fc082e8583771ea01ea87ff2782d2d2b073ff900c2185d66db3d5a906b9: Status 404 returned error can't find the container with id 786e2fc082e8583771ea01ea87ff2782d2d2b073ff900c2185d66db3d5a906b9 Nov 02 15:22:45.464114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.464082 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.464183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.464147 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.464183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.464180 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.502173 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.502149 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:22:45.685956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.685903 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" event={"ID":"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00","Type":"ContainerStarted","Data":"87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b"} Nov 02 15:22:45.686458 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.686028 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" event={"ID":"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00","Type":"ContainerStarted","Data":"786e2fc082e8583771ea01ea87ff2782d2d2b073ff900c2185d66db3d5a906b9"} Nov 02 15:22:45.688100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.688077 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" event={"ID":"1d224219-f041-4188-a745-e0bf2e46f1b4","Type":"ContainerStarted","Data":"c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3"} Nov 02 15:22:45.688227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.688114 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" event={"ID":"1d224219-f041-4188-a745-e0bf2e46f1b4","Type":"ContainerStarted","Data":"aafd28db9bb587e605c8cdbdacafda671c8b02a841f77c7cabde56f27184ab52"} Nov 02 15:22:45.802237 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.802119 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" podStartSLOduration=1.802106297 podStartE2EDuration="1.802106297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:45.801809624 +0000 UTC m=+1469.475012821" watchObservedRunningTime="2026-11-02 15:22:45.802106297 +0000 UTC m=+1469.475309493" Nov 02 15:22:45.827350 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.827315 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:45.949000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:45.948568 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" podStartSLOduration=1.9485449099999999 podStartE2EDuration="1.94854491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:45.92231317 +0000 UTC m=+1469.595516366" watchObservedRunningTime="2026-11-02 15:22:45.94854491 +0000 UTC m=+1469.621748106" Nov 02 15:22:46.084052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:46.083889 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:46.086470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:46.086100 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:22:46.086470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:46.086161 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 15:22:46.836017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:46.829993 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:22:46.836017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:46.830074 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 15:22:47.006306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:47.006265 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:47.090053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:47.083371 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:22:47.090053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:47.083445 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 15:22:47.839999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:47.837334 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:22:47.839999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:47.837398 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 15:22:48.081994 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:48.081947 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:22:48.082166 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:48.082062 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 15:22:48.840999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:48.836408 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:22:48.840999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:48.836477 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 15:22:49.082367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:49.082334 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:22:49.082555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:49.082393 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 15:22:49.369962 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:49.369918 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:49.513146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:49.513104 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:49.667659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:49.667625 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:22:49.672470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:49.672428 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:49.836001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:49.828394 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:22:49.836001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:49.828473 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 15:22:50.082412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:50.082332 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:22:50.082867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:50.082838 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 15:22:50.765813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:50.765765 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:50.841008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:50.840068 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:22:50.841008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:50.840134 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 15:22:50.938928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:50.938865 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:51.082144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:51.081903 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:22:51.082144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:51.082002 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 15:22:51.827593 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:51.827559 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:22:51.828020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:51.827613 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 15:22:51.960866 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:51.960829 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:52.081754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:52.081663 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:22:52.081754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:52.081720 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 15:22:52.354025 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:52.353910 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:52.929242 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:52.929205 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:52.930002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:52.929964 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:52.931416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:52.931398 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:22:53.094969 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:53.094924 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:53.095564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:53.095542 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:53.096938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:53.096909 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:22:55.799434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:55.799391 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:56.509636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:56.509596 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:57.037493 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:57.037454 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:57.771830 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:57.771796 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:22:58.233465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:22:58.233414 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:00.512961 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:00.506662 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:00.658634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:00.658458 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:03.092900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.092867 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd"] Nov 02 15:23:03.098711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.098687 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.149903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.149865 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a758f7bf-serving-cert\"" Nov 02 15:23:03.202065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.202030 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd"] Nov 02 15:23:03.208901 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.208867 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qplhg\" (UniqueName: \"kubernetes.io/projected/db483859-9ded-4eee-b161-aec1d60c71e7-kube-api-access-qplhg\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.209447 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.209388 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.209627 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.209448 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-metrics-certs\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.209720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.209653 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db483859-9ded-4eee-b161-aec1d60c71e7-openshift-service-ca\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.295176 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.295141 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:03.311401 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.311368 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qplhg\" (UniqueName: \"kubernetes.io/projected/db483859-9ded-4eee-b161-aec1d60c71e7-kube-api-access-qplhg\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.311569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.311525 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.311569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.311548 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-metrics-certs\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.311669 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.311652 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db483859-9ded-4eee-b161-aec1d60c71e7-openshift-service-ca\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.311720 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:23:03.311676 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-serving-cert: secret "maas-api-e2e-ait-a758f7bf-serving-cert" not found Nov 02 15:23:03.311757 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:23:03.311741 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls podName:db483859-9ded-4eee-b161-aec1d60c71e7 nodeName:}" failed. No retries permitted until 2026-11-02 15:23:03.811721239 +0000 UTC m=+1487.484924435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls") pod "maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" (UID: "db483859-9ded-4eee-b161-aec1d60c71e7") : secret "maas-api-e2e-ait-a758f7bf-serving-cert" not found Nov 02 15:23:03.312390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.312358 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db483859-9ded-4eee-b161-aec1d60c71e7-openshift-service-ca\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.314190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.314167 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-metrics-certs\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.409949 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.409912 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qplhg\" (UniqueName: \"kubernetes.io/projected/db483859-9ded-4eee-b161-aec1d60c71e7-kube-api-access-qplhg\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.508019 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.507957 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:03.661235 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.660826 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:03.808227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.808189 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:03.841100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.841073 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:03.843730 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:03.843707 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls\") pod \"maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:04.025047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:04.022308 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:04.135036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:04.133581 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:04.960390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:04.960359 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd"] Nov 02 15:23:04.961335 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:23:04.961305 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb483859_9ded_4eee_b161_aec1d60c71e7.slice/crio-4dec95de81bd1fa7185d134a339547e81815dc42c956c26e515c808136b08f86 WatchSource:0}: Error finding container 4dec95de81bd1fa7185d134a339547e81815dc42c956c26e515c808136b08f86: Status 404 returned error can't find the container with id 4dec95de81bd1fa7185d134a339547e81815dc42c956c26e515c808136b08f86 Nov 02 15:23:04.963283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:04.963262 5433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:23:05.616102 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:05.616069 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" event={"ID":"db483859-9ded-4eee-b161-aec1d60c71e7","Type":"ContainerStarted","Data":"4dec95de81bd1fa7185d134a339547e81815dc42c956c26e515c808136b08f86"} Nov 02 15:23:06.624618 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:06.624580 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" event={"ID":"db483859-9ded-4eee-b161-aec1d60c71e7","Type":"ContainerStarted","Data":"27ad989facb1859c535baa2de02ab1642ccb03c4545b32a93c8f9e072d5a2366"} Nov 02 15:23:06.625090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:06.624683 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:06.746722 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:06.746666 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" podStartSLOduration=2.846269672 podStartE2EDuration="3.746651352s" totalImagesPullingTime="900.38168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:06.743176543 +0000 UTC m=+1490.416379739" watchObservedRunningTime="2026-11-02 15:23:06.746651352 +0000 UTC m=+1490.419854547" Nov 02 15:23:09.373073 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:09.373033 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:10.624292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:10.624258 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:11.015167 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:23:11.015077 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:23:12.638224 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:12.638195 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:23:14.465201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:14.465167 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:19.391874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:19.391838 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:19.589044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:19.588337 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:19.743153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:19.743053 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:23.771845 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:23.771811 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:23.928087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:23.928055 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:24.213269 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:24.213236 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:24.621462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:24.621378 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:24.819534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:24.819503 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:30.055274 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:30.055238 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:30.259885 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:30.259830 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:30.434080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:30.433802 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:31.104929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:31.104835 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:31.278723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:31.278689 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:32.000001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:31.999320 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:23:32.044281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:32.044235 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.092998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:32.092423 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.438105 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:32.438057 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.446105 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:32.446049 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.572329 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:32.572285 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.991996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:32.991061 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:23:33.010999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.002430 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:33.010999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.002944 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:23:33.054128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.052259 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:23:33.054128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.052845 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:23:33.107995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.107878 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 02 15:23:33.145001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.119417 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:23:33.509136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.509089 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:23:33.512302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.512275 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:23:33.579829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.579786 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:33.587436 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:33.580659 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:23:34.017691 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.017641 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:34.040283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.040177 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:34.052068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.041083 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:23:34.053742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.052574 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:34.143268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.121609 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:34.544142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.524574 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:34.573459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.573429 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:23:34.573719 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.573707 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:23:34.588162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.588106 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 15:23:34.676633 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.673964 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:23:34.705377 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.705347 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:23:34.721119 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.721089 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:23:34.729478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:34.729450 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:23:36.075198 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:36.075164 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:41.013302 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:23:41.013260 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:23:41.261425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:41.261387 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:46.364599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:46.364558 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:46.542265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:46.542225 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:51.399241 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:51.399198 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:54.910501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:54.910466 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv"] Nov 02 15:23:54.914861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:54.914842 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.006023 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.005948 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv"] Nov 02 15:23:55.021684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.021649 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.021886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.021786 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.021886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.021865 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzfxz\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-kube-api-access-pzfxz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.022050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.021895 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.022050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.022013 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.022175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.022141 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.022353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.022333 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.022436 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.022383 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.022513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.022436 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9d0b06-cc52-424e-85b4-9d53582d127c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.123741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.123702 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.123958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.123775 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.123958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.123821 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.123958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.123874 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9d0b06-cc52-424e-85b4-9d53582d127c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.124259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.124051 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.124259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.124173 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.124259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.124248 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzfxz\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-kube-api-access-pzfxz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.124419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.124283 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.124419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.124383 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.124537 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.124490 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.124743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.124719 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.124893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.124813 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.124893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.124862 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9d0b06-cc52-424e-85b4-9d53582d127c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.125113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.125095 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.126393 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.126366 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.126965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.126948 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.181714 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.181635 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.181848 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.181758 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzfxz\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-kube-api-access-pzfxz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jhvv\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.230371 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.230333 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:55.654962 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.654938 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv"] Nov 02 15:23:55.656237 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:23:55.656203 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9d0b06_cc52_424e_85b4_9d53582d127c.slice/crio-c7346fba3787e456b81d0775a87e9177e7b109b52144fd364a62ea6dec6cc59e WatchSource:0}: Error finding container c7346fba3787e456b81d0775a87e9177e7b109b52144fd364a62ea6dec6cc59e: Status 404 returned error can't find the container with id c7346fba3787e456b81d0775a87e9177e7b109b52144fd364a62ea6dec6cc59e Nov 02 15:23:55.659040 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.658999 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:23:55.659133 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.659079 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:23:55.659133 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:55.659116 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:23:56.458746 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:56.458668 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:23:56.624000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:56.620845 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" event={"ID":"ce9d0b06-cc52-424e-85b4-9d53582d127c","Type":"ContainerStarted","Data":"d3f205f7d8d9b5ff08a512640270d7ab76d7f4a4755fb8aab492065c1eab4478"} Nov 02 15:23:56.624000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:56.620920 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" event={"ID":"ce9d0b06-cc52-424e-85b4-9d53582d127c","Type":"ContainerStarted","Data":"c7346fba3787e456b81d0775a87e9177e7b109b52144fd364a62ea6dec6cc59e"} Nov 02 15:23:56.842451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:56.842311 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podStartSLOduration=2.842296554 podStartE2EDuration="2.842296554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:56.791202356 +0000 UTC m=+1540.464405552" watchObservedRunningTime="2026-11-02 15:23:56.842296554 +0000 UTC m=+1540.515499749" Nov 02 15:23:57.241212 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:57.241170 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:23:57.242921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:57.242891 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:23:57.243078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:57.242950 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 15:23:58.237999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:58.233383 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:23:58.237999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:58.233448 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 15:23:59.233000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:59.231434 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:23:59.233000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:23:59.231502 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 15:24:00.249483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:00.249448 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:24:00.250097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:00.250065 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 15:24:01.236001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:01.234507 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:24:01.236001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:01.234595 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 15:24:02.283999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:02.268790 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:02.283999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:02.268929 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:24:02.283999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:02.269008 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 15:24:02.427000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:02.419359 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:02.456815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:02.456778 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:02.592194 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:02.592097 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:02.769121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:02.769075 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:02.777432 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:02.777346 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:03.143835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:03.143779 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:03.236743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:03.236312 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:24:03.236743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:03.236374 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 15:24:04.231380 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:04.231347 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:24:04.231763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:04.231411 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 15:24:05.243009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:05.240406 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:24:05.243009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:05.240488 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 15:24:06.417694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:06.417658 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:24:06.418725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:06.418690 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:24:06.419509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:06.419487 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:24:08.011288 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:08.011238 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:11.013410 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:24:11.013374 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:24:13.376419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:13.376371 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:19.055362 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:19.055320 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:22.137271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:22.137225 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:22.313875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:22.313779 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:22.931164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:22.931115 5433 generic.go:356] "Generic (PLEG): container finished" podID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerID="b057dd3dbacc95a3a0616ccda8ca6e3c7c0c6503486e6834ffeedaf45132e3cd" exitCode=0 Nov 02 15:24:22.931350 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:22.931189 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" event={"ID":"4c40bb21-2237-4cb8-8792-9e3dd99d62f7","Type":"ContainerDied","Data":"b057dd3dbacc95a3a0616ccda8ca6e3c7c0c6503486e6834ffeedaf45132e3cd"} Nov 02 15:24:22.931733 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:22.931716 5433 scope.go:122] "RemoveContainer" containerID="b057dd3dbacc95a3a0616ccda8ca6e3c7c0c6503486e6834ffeedaf45132e3cd" Nov 02 15:24:22.933461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:22.933434 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:22.933532 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:22.933505 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:22.933574 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:22.933538 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:23.956676 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:23.956634 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" event={"ID":"4c40bb21-2237-4cb8-8792-9e3dd99d62f7","Type":"ContainerStarted","Data":"5ecbd1337de1755ae7defc865f7780a36895d7ab532bf07d8cabb6ad8f8415ed"} Nov 02 15:24:24.291220 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:24.291134 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:24.742051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:24.741796 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:24:24.742051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:24.741829 5433 patch_prober.go:29] interesting pod/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:24:24.742051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:24.741882 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 15:24:24.941871 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:24.941831 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:25.741460 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:25.741414 5433 patch_prober.go:29] interesting pod/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:24:25.742009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:25.741480 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 15:24:26.742082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:26.742045 5433 patch_prober.go:29] interesting pod/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:24:26.742653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:26.742625 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 15:24:27.744036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:27.743111 5433 patch_prober.go:29] interesting pod/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:24:27.744036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:27.743172 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 15:24:27.889560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:27.889519 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz"] Nov 02 15:24:28.313802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:28.313761 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:28.498001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:28.492480 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:28.743332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:28.743284 5433 patch_prober.go:29] interesting pod/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:24:28.765127 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:28.765081 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 15:24:29.764326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:29.764294 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:24:29.766230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:29.766209 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:24:29.779204 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:29.779167 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:24:30.413987 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:30.413945 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:32.890694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:32.890630 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" podUID="1d3469de-4872-4276-ab00-96856badbde3" containerName="payload-processing" containerID="cri-o://1b6ab45cc0f1503ce0a8e0b702059c91b1d1dac00abc2c59207fd10835ee9632" gracePeriod=25 Nov 02 15:24:33.047968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.047933 5433 generic.go:356] "Generic (PLEG): container finished" podID="1d3469de-4872-4276-ab00-96856badbde3" containerID="1b6ab45cc0f1503ce0a8e0b702059c91b1d1dac00abc2c59207fd10835ee9632" exitCode=0 Nov 02 15:24:33.048149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.048022 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" event={"ID":"1d3469de-4872-4276-ab00-96856badbde3","Type":"ContainerDied","Data":"1b6ab45cc0f1503ce0a8e0b702059c91b1d1dac00abc2c59207fd10835ee9632"} Nov 02 15:24:33.054520 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.054492 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:33.173624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.173601 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:24:33.260278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.260241 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3469de-4872-4276-ab00-96856badbde3-kube-api-access-xrg54\" (UniqueName: \"kubernetes.io/projected/1d3469de-4872-4276-ab00-96856badbde3-kube-api-access-xrg54\") pod \"1d3469de-4872-4276-ab00-96856badbde3\" (UID: \"1d3469de-4872-4276-ab00-96856badbde3\") " Nov 02 15:24:33.260464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.260453 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3469de-4872-4276-ab00-96856badbde3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3469de-4872-4276-ab00-96856badbde3-plugins-config-volume\") pod \"1d3469de-4872-4276-ab00-96856badbde3\" (UID: \"1d3469de-4872-4276-ab00-96856badbde3\") " Nov 02 15:24:33.260800 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.260777 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3469de-4872-4276-ab00-96856badbde3-plugins-config-volume" pod "1d3469de-4872-4276-ab00-96856badbde3" (UID: "1d3469de-4872-4276-ab00-96856badbde3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:33.260949 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.260933 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3469de-4872-4276-ab00-96856badbde3-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:33.262455 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.262424 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3469de-4872-4276-ab00-96856badbde3-kube-api-access-xrg54" pod "1d3469de-4872-4276-ab00-96856badbde3" (UID: "1d3469de-4872-4276-ab00-96856badbde3"). InnerVolumeSpecName "kube-api-access-xrg54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:33.362584 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:33.362551 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrg54\" (UniqueName: \"kubernetes.io/projected/1d3469de-4872-4276-ab00-96856badbde3-kube-api-access-xrg54\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:34.056148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:34.056115 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" event={"ID":"1d3469de-4872-4276-ab00-96856badbde3","Type":"ContainerDied","Data":"2c17ff1c3c50e30ae835be8ff02777e38d65181c0fc584e46ba098a33a0293dc"} Nov 02 15:24:34.056565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:34.056154 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz" Nov 02 15:24:34.056565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:34.056177 5433 scope.go:122] "RemoveContainer" containerID="1b6ab45cc0f1503ce0a8e0b702059c91b1d1dac00abc2c59207fd10835ee9632" Nov 02 15:24:34.214502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:34.214465 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz"] Nov 02 15:24:34.259670 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:34.259637 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a758f7bf-c76c67b99-hgrfz"] Nov 02 15:24:34.457745 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:34.457712 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3469de-4872-4276-ab00-96856badbde3" path="/var/lib/kubelet/pods/1d3469de-4872-4276-ab00-96856badbde3/volumes" Nov 02 15:24:35.876135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:35.876102 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:36.658457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:36.658415 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:36.801987 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:36.801939 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:40.695005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:40.692910 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:41.064293 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:24:41.064194 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:24:43.420187 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:43.420152 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7"] Nov 02 15:24:43.524347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:43.524312 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d"] Nov 02 15:24:43.781806 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:43.781728 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:43.984996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:43.981551 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:45.556145 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:45.556107 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:46.859545 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:46.859509 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:48.420849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.420786 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" podUID="81b025e5-4c21-4036-91b2-c97f853feee9" containerName="payload-processing" containerID="cri-o://7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184" gracePeriod=25 Nov 02 15:24:48.524861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.524806 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" podUID="ea639834-f676-4602-8ea3-265b074e2d10" containerName="payload-pre-processing" containerID="cri-o://82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3" gracePeriod=25 Nov 02 15:24:48.736662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.736631 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:24:48.826066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.826032 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b025e5-4c21-4036-91b2-c97f853feee9-kube-api-access-9pfb4\" (UniqueName: \"kubernetes.io/projected/81b025e5-4c21-4036-91b2-c97f853feee9-kube-api-access-9pfb4\") pod \"81b025e5-4c21-4036-91b2-c97f853feee9\" (UID: \"81b025e5-4c21-4036-91b2-c97f853feee9\") " Nov 02 15:24:48.826276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.826258 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b025e5-4c21-4036-91b2-c97f853feee9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b025e5-4c21-4036-91b2-c97f853feee9-plugins-config-volume\") pod \"81b025e5-4c21-4036-91b2-c97f853feee9\" (UID: \"81b025e5-4c21-4036-91b2-c97f853feee9\") " Nov 02 15:24:48.826856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.826823 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b025e5-4c21-4036-91b2-c97f853feee9-plugins-config-volume" pod "81b025e5-4c21-4036-91b2-c97f853feee9" (UID: "81b025e5-4c21-4036-91b2-c97f853feee9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:48.827012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.826999 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b025e5-4c21-4036-91b2-c97f853feee9-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:48.829189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.829168 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b025e5-4c21-4036-91b2-c97f853feee9-kube-api-access-9pfb4" pod "81b025e5-4c21-4036-91b2-c97f853feee9" (UID: "81b025e5-4c21-4036-91b2-c97f853feee9"). InnerVolumeSpecName "kube-api-access-9pfb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:48.833558 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.833540 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:24:48.928147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.928114 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea639834-f676-4602-8ea3-265b074e2d10-kube-api-access-q6c89\" (UniqueName: \"kubernetes.io/projected/ea639834-f676-4602-8ea3-265b074e2d10-kube-api-access-q6c89\") pod \"ea639834-f676-4602-8ea3-265b074e2d10\" (UID: \"ea639834-f676-4602-8ea3-265b074e2d10\") " Nov 02 15:24:48.928344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.928209 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea639834-f676-4602-8ea3-265b074e2d10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea639834-f676-4602-8ea3-265b074e2d10-plugins-config-volume\") pod \"ea639834-f676-4602-8ea3-265b074e2d10\" (UID: \"ea639834-f676-4602-8ea3-265b074e2d10\") " Nov 02 15:24:48.928578 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.928552 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea639834-f676-4602-8ea3-265b074e2d10-plugins-config-volume" pod "ea639834-f676-4602-8ea3-265b074e2d10" (UID: "ea639834-f676-4602-8ea3-265b074e2d10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:48.928897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.928882 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfb4\" (UniqueName: \"kubernetes.io/projected/81b025e5-4c21-4036-91b2-c97f853feee9-kube-api-access-9pfb4\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:48.928945 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.928899 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea639834-f676-4602-8ea3-265b074e2d10-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:48.930299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:48.930276 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea639834-f676-4602-8ea3-265b074e2d10-kube-api-access-q6c89" pod "ea639834-f676-4602-8ea3-265b074e2d10" (UID: "ea639834-f676-4602-8ea3-265b074e2d10"). InnerVolumeSpecName "kube-api-access-q6c89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:49.030164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.030084 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6c89\" (UniqueName: \"kubernetes.io/projected/ea639834-f676-4602-8ea3-265b074e2d10-kube-api-access-q6c89\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:49.268475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.268436 5433 generic.go:356] "Generic (PLEG): container finished" podID="81b025e5-4c21-4036-91b2-c97f853feee9" containerID="7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184" exitCode=0 Nov 02 15:24:49.268689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.268488 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" Nov 02 15:24:49.268689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.268526 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" event={"ID":"81b025e5-4c21-4036-91b2-c97f853feee9","Type":"ContainerDied","Data":"7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184"} Nov 02 15:24:49.268689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.268597 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7" event={"ID":"81b025e5-4c21-4036-91b2-c97f853feee9","Type":"ContainerDied","Data":"ac07daf977ea82c248a55bea7ded2a1f4a3e01a649f4afbb9fcdde94b6821f8b"} Nov 02 15:24:49.268689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.268626 5433 scope.go:122] "RemoveContainer" containerID="7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184" Nov 02 15:24:49.270417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.270388 5433 generic.go:356] "Generic (PLEG): container finished" podID="ea639834-f676-4602-8ea3-265b074e2d10" containerID="82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3" exitCode=0 Nov 02 15:24:49.270558 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.270438 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" Nov 02 15:24:49.270558 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.270473 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" event={"ID":"ea639834-f676-4602-8ea3-265b074e2d10","Type":"ContainerDied","Data":"82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3"} Nov 02 15:24:49.270558 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.270527 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d" event={"ID":"ea639834-f676-4602-8ea3-265b074e2d10","Type":"ContainerDied","Data":"ede9db9b63f9df509f2cb0b3ebaac88f51c0b8b78232ed18835e96325eb9fd1e"} Nov 02 15:24:49.280210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.280106 5433 scope.go:122] "RemoveContainer" containerID="7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184" Nov 02 15:24:49.280647 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:24:49.280622 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184\": container with ID starting with 7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184 not found: ID does not exist" containerID="7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184" Nov 02 15:24:49.280747 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.280657 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184"} err="failed to get container status \"7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184\": rpc error: code = NotFound desc = could not find container \"7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184\": container with ID starting with 7357e4d0e59e2cb301b7cf9787ee96ba9eb98888e61467b9baff2a2414e84184 not found: ID does not exist" Nov 02 15:24:49.280747 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.280679 5433 scope.go:122] "RemoveContainer" containerID="82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3" Nov 02 15:24:49.292560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.292538 5433 scope.go:122] "RemoveContainer" containerID="82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3" Nov 02 15:24:49.292810 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:24:49.292793 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3\": container with ID starting with 82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3 not found: ID does not exist" containerID="82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3" Nov 02 15:24:49.292863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.292818 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3"} err="failed to get container status \"82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3\": rpc error: code = NotFound desc = could not find container \"82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3\": container with ID starting with 82e3c3029f93768fd2fb8d97a361aa20345d86b8b66ce8fbc967b31f73efe6c3 not found: ID does not exist" Nov 02 15:24:49.425020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.424984 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7"] Nov 02 15:24:49.430169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.430138 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd9398b4-754c585b5-cvqr7"] Nov 02 15:24:49.531961 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.531877 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d"] Nov 02 15:24:49.535394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:49.535372 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd9398b4-749f9bf64c-88k9d"] Nov 02 15:24:50.458856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:50.458820 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b025e5-4c21-4036-91b2-c97f853feee9" path="/var/lib/kubelet/pods/81b025e5-4c21-4036-91b2-c97f853feee9/volumes" Nov 02 15:24:50.459282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:50.459266 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea639834-f676-4602-8ea3-265b074e2d10" path="/var/lib/kubelet/pods/ea639834-f676-4602-8ea3-265b074e2d10/volumes" Nov 02 15:24:52.018035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:52.017966 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:52.460804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:52.460766 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:57.496143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:57.496103 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns"] Nov 02 15:24:57.777084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:57.776962 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:57.976748 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:57.976706 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:24:58.134999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:24:58.134088 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:02.496551 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:02.496498 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" podUID="a262cd1f-20d5-40ee-82cc-d669bcbf8b42" containerName="payload-processing" containerID="cri-o://6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa" gracePeriod=25 Nov 02 15:25:02.790127 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:02.790101 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:25:02.939542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:02.939504 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-plugins-config-volume\") pod \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\" (UID: \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\") " Nov 02 15:25:02.939758 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:02.939651 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-kube-api-access-7svsj\" (UniqueName: \"kubernetes.io/projected/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-kube-api-access-7svsj\") pod \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\" (UID: \"a262cd1f-20d5-40ee-82cc-d669bcbf8b42\") " Nov 02 15:25:02.939997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:02.939941 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-plugins-config-volume" pod "a262cd1f-20d5-40ee-82cc-d669bcbf8b42" (UID: "a262cd1f-20d5-40ee-82cc-d669bcbf8b42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:02.940169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:02.940151 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:02.941809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:02.941777 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-kube-api-access-7svsj" pod "a262cd1f-20d5-40ee-82cc-d669bcbf8b42" (UID: "a262cd1f-20d5-40ee-82cc-d669bcbf8b42"). InnerVolumeSpecName "kube-api-access-7svsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:03.042298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.042220 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svsj\" (UniqueName: \"kubernetes.io/projected/a262cd1f-20d5-40ee-82cc-d669bcbf8b42-kube-api-access-7svsj\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:03.159008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.158944 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:03.387521 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.387489 5433 generic.go:356] "Generic (PLEG): container finished" podID="a262cd1f-20d5-40ee-82cc-d669bcbf8b42" containerID="6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa" exitCode=0 Nov 02 15:25:03.387703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.387547 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" Nov 02 15:25:03.387703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.387574 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" event={"ID":"a262cd1f-20d5-40ee-82cc-d669bcbf8b42","Type":"ContainerDied","Data":"6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa"} Nov 02 15:25:03.387703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.387633 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns" event={"ID":"a262cd1f-20d5-40ee-82cc-d669bcbf8b42","Type":"ContainerDied","Data":"0cab960e18dd3762bb6700b6e6eb54aae283063c75e156c577b43961522f2c60"} Nov 02 15:25:03.387703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.387666 5433 scope.go:122] "RemoveContainer" containerID="6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa" Nov 02 15:25:03.399455 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.399437 5433 scope.go:122] "RemoveContainer" containerID="6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa" Nov 02 15:25:03.399713 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:25:03.399695 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa\": container with ID starting with 6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa not found: ID does not exist" containerID="6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa" Nov 02 15:25:03.399760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.399721 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa"} err="failed to get container status \"6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa\": rpc error: code = NotFound desc = could not find container \"6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa\": container with ID starting with 6f6216f1bb186c9dd7532bc50ea27e866d9ae6bd26b6e1cf62dd8f2e22bc48aa not found: ID does not exist" Nov 02 15:25:03.479394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.479351 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:03.545856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.545826 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns"] Nov 02 15:25:03.592789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.592750 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-06a527d4-fb7485dcf-p6zns"] Nov 02 15:25:03.679916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:03.679827 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:04.459290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:04.459249 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a262cd1f-20d5-40ee-82cc-d669bcbf8b42" path="/var/lib/kubelet/pods/a262cd1f-20d5-40ee-82cc-d669bcbf8b42/volumes" Nov 02 15:25:08.200758 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:08.200723 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:11.013787 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:25:11.013748 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:25:12.523814 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:12.523760 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd"] Nov 02 15:25:12.525047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:12.525009 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" podUID="db483859-9ded-4eee-b161-aec1d60c71e7" containerName="maas-api" containerID="cri-o://27ad989facb1859c535baa2de02ab1642ccb03c4545b32a93c8f9e072d5a2366" gracePeriod=30 Nov 02 15:25:12.649761 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:12.629586 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:12.674407 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:12.674025 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" podUID="db483859-9ded-4eee-b161-aec1d60c71e7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.87:8443/health\": dial tcp 10.133.0.87:8443: connect: connection refused" Nov 02 15:25:13.462052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.462013 5433 generic.go:356] "Generic (PLEG): container finished" podID="db483859-9ded-4eee-b161-aec1d60c71e7" containerID="27ad989facb1859c535baa2de02ab1642ccb03c4545b32a93c8f9e072d5a2366" exitCode=0 Nov 02 15:25:13.462247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.462046 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" event={"ID":"db483859-9ded-4eee-b161-aec1d60c71e7","Type":"ContainerDied","Data":"27ad989facb1859c535baa2de02ab1642ccb03c4545b32a93c8f9e072d5a2366"} Nov 02 15:25:13.462332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.462313 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" event={"ID":"db483859-9ded-4eee-b161-aec1d60c71e7","Type":"ContainerDied","Data":"4dec95de81bd1fa7185d134a339547e81815dc42c956c26e515c808136b08f86"} Nov 02 15:25:13.462396 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.462349 5433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dec95de81bd1fa7185d134a339547e81815dc42c956c26e515c808136b08f86" Nov 02 15:25:13.473409 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.473347 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:25:13.598879 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.598838 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls\") pod \"db483859-9ded-4eee-b161-aec1d60c71e7\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " Nov 02 15:25:13.599412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.598995 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db483859-9ded-4eee-b161-aec1d60c71e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db483859-9ded-4eee-b161-aec1d60c71e7-openshift-service-ca\") pod \"db483859-9ded-4eee-b161-aec1d60c71e7\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " Nov 02 15:25:13.599412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.599034 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-metrics-certs\") pod \"db483859-9ded-4eee-b161-aec1d60c71e7\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " Nov 02 15:25:13.599412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.599100 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db483859-9ded-4eee-b161-aec1d60c71e7-kube-api-access-qplhg\" (UniqueName: \"kubernetes.io/projected/db483859-9ded-4eee-b161-aec1d60c71e7-kube-api-access-qplhg\") pod \"db483859-9ded-4eee-b161-aec1d60c71e7\" (UID: \"db483859-9ded-4eee-b161-aec1d60c71e7\") " Nov 02 15:25:13.600870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.600817 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db483859-9ded-4eee-b161-aec1d60c71e7-openshift-service-ca" pod "db483859-9ded-4eee-b161-aec1d60c71e7" (UID: "db483859-9ded-4eee-b161-aec1d60c71e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:13.606499 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.606455 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db483859-9ded-4eee-b161-aec1d60c71e7-kube-api-access-qplhg" pod "db483859-9ded-4eee-b161-aec1d60c71e7" (UID: "db483859-9ded-4eee-b161-aec1d60c71e7"). InnerVolumeSpecName "kube-api-access-qplhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:13.606899 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.606874 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls" pod "db483859-9ded-4eee-b161-aec1d60c71e7" (UID: "db483859-9ded-4eee-b161-aec1d60c71e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:25:13.608107 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.608082 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-metrics-certs" pod "db483859-9ded-4eee-b161-aec1d60c71e7" (UID: "db483859-9ded-4eee-b161-aec1d60c71e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:25:13.702906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.702870 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:13.702906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.702905 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db483859-9ded-4eee-b161-aec1d60c71e7-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:13.702906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.702915 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db483859-9ded-4eee-b161-aec1d60c71e7-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:13.703209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:13.702924 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qplhg\" (UniqueName: \"kubernetes.io/projected/db483859-9ded-4eee-b161-aec1d60c71e7-kube-api-access-qplhg\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:14.472570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:14.472545 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd" Nov 02 15:25:14.496357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:14.496326 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:14.687109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:14.687073 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd"] Nov 02 15:25:14.690761 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:14.690734 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a758f7bf-7fd4b698c9-rj9fd"] Nov 02 15:25:14.855609 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:14.855500 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:15.159947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:15.159896 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:15.441243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:15.441156 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg"] Nov 02 15:25:15.441917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:15.441887 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" containerID="cri-o://5ecbd1337de1755ae7defc865f7780a36895d7ab532bf07d8cabb6ad8f8415ed" gracePeriod=30 Nov 02 15:25:15.472001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:15.465615 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:16.039647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:16.038431 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:16.463579 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:16.463545 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db483859-9ded-4eee-b161-aec1d60c71e7" path="/var/lib/kubelet/pods/db483859-9ded-4eee-b161-aec1d60c71e7/volumes" Nov 02 15:25:17.090819 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.090738 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl"] Nov 02 15:25:17.091528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091509 5433 cpu_manager.go:414] "containerMap: removing container" podUID="a262cd1f-20d5-40ee-82cc-d669bcbf8b42" containerName="payload-processing" Nov 02 15:25:17.091570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091529 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a262cd1f-20d5-40ee-82cc-d669bcbf8b42" containerName="payload-processing" Nov 02 15:25:17.091570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091557 5433 cpu_manager.go:414] "containerMap: removing container" podUID="81b025e5-4c21-4036-91b2-c97f853feee9" containerName="payload-processing" Nov 02 15:25:17.091570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091563 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b025e5-4c21-4036-91b2-c97f853feee9" containerName="payload-processing" Nov 02 15:25:17.091672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091579 5433 cpu_manager.go:414] "containerMap: removing container" podUID="1d3469de-4872-4276-ab00-96856badbde3" containerName="payload-processing" Nov 02 15:25:17.091672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091584 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3469de-4872-4276-ab00-96856badbde3" containerName="payload-processing" Nov 02 15:25:17.091672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091592 5433 cpu_manager.go:414] "containerMap: removing container" podUID="ea639834-f676-4602-8ea3-265b074e2d10" containerName="payload-pre-processing" Nov 02 15:25:17.091672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091598 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea639834-f676-4602-8ea3-265b074e2d10" containerName="payload-pre-processing" Nov 02 15:25:17.091672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091608 5433 cpu_manager.go:414] "containerMap: removing container" podUID="db483859-9ded-4eee-b161-aec1d60c71e7" containerName="maas-api" Nov 02 15:25:17.091672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091613 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db483859-9ded-4eee-b161-aec1d60c71e7" containerName="maas-api" Nov 02 15:25:17.091853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091714 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b025e5-4c21-4036-91b2-c97f853feee9" containerName="payload-processing" Nov 02 15:25:17.091853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091725 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3469de-4872-4276-ab00-96856badbde3" containerName="payload-processing" Nov 02 15:25:17.091853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091731 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea639834-f676-4602-8ea3-265b074e2d10" containerName="payload-pre-processing" Nov 02 15:25:17.091853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091737 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="db483859-9ded-4eee-b161-aec1d60c71e7" containerName="maas-api" Nov 02 15:25:17.091853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.091747 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a262cd1f-20d5-40ee-82cc-d669bcbf8b42" containerName="payload-processing" Nov 02 15:25:17.096602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.096581 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.188063 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.188034 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9d0b05fd-openshift-default-dockercfg-qzqs7\"" Nov 02 15:25:17.195246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.195209 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:17.198773 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.198744 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl"] Nov 02 15:25:17.296458 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.296422 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb5qs\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-kube-api-access-fb5qs\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.296892 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.296851 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-credential-socket\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.297091 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.297074 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8622a11-4189-477c-bd43-1cc6d12250ca-istiod-ca-cert\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.303776 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.303749 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-data\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.304118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.304082 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-podinfo\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.304291 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.304270 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-token\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.304484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.304418 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-socket\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.304693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.304673 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-envoy\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.304888 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.304869 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-certs\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.416135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.408102 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-socket\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.416135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.408216 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-envoy\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.416135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.408300 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-certs\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.416135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.408411 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5qs\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-kube-api-access-fb5qs\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.416135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.408509 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-credential-socket\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.424111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.423665 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8622a11-4189-477c-bd43-1cc6d12250ca-istiod-ca-cert\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.424111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.423801 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-data\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.424111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.424021 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-podinfo\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.424111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.424057 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-token\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.424801 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.424767 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-certs\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.428538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.425338 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-credential-socket\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.428538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.426776 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-data\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.428844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.428814 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-socket\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.440039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.435308 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8622a11-4189-477c-bd43-1cc6d12250ca-istiod-ca-cert\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.440039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.437285 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-envoy\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.440039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.439924 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-podinfo\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.497001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.493527 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:17.514029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.505749 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb5qs\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-kube-api-access-fb5qs\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.555001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.550182 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-token\") pod \"e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:17.710599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:17.710520 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:18.161582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:18.161553 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl"] Nov 02 15:25:18.162101 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:25:18.162075 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8622a11_4189_477c_bd43_1cc6d12250ca.slice/crio-99c97e2e7b47606ae39fc8a0a44938a71e13153b9fbb9d6d66078b2f71f20249 WatchSource:0}: Error finding container 99c97e2e7b47606ae39fc8a0a44938a71e13153b9fbb9d6d66078b2f71f20249: Status 404 returned error can't find the container with id 99c97e2e7b47606ae39fc8a0a44938a71e13153b9fbb9d6d66078b2f71f20249 Nov 02 15:25:18.164695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:18.164662 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:18.164754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:18.164727 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:18.164798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:18.164759 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:18.604835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:18.604733 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" event={"ID":"c8622a11-4189-477c-bd43-1cc6d12250ca","Type":"ContainerStarted","Data":"066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce"} Nov 02 15:25:18.604835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:18.604800 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" event={"ID":"c8622a11-4189-477c-bd43-1cc6d12250ca","Type":"ContainerStarted","Data":"99c97e2e7b47606ae39fc8a0a44938a71e13153b9fbb9d6d66078b2f71f20249"} Nov 02 15:25:18.710587 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:18.710558 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:18.794363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:18.794311 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" podStartSLOduration=1.794296237 podStartE2EDuration="1.794296237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:18.764072211 +0000 UTC m=+1622.437275412" watchObservedRunningTime="2026-11-02 15:25:18.794296237 +0000 UTC m=+1622.467499431" Nov 02 15:25:19.717839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:19.717806 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:20.621500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.621459 5433 generic.go:356] "Generic (PLEG): container finished" podID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerID="5ecbd1337de1755ae7defc865f7780a36895d7ab532bf07d8cabb6ad8f8415ed" exitCode=0 Nov 02 15:25:20.621693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.621538 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" event={"ID":"4c40bb21-2237-4cb8-8792-9e3dd99d62f7","Type":"ContainerDied","Data":"5ecbd1337de1755ae7defc865f7780a36895d7ab532bf07d8cabb6ad8f8415ed"} Nov 02 15:25:20.621693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.621611 5433 scope.go:122] "RemoveContainer" containerID="b057dd3dbacc95a3a0616ccda8ca6e3c7c0c6503486e6834ffeedaf45132e3cd" Nov 02 15:25:20.622062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.622040 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:20.623229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.623196 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:25:20.751200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.751176 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:25:20.889746 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.889709 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-credential-socket\") pod \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " Nov 02 15:25:20.889921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.889763 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-data\") pod \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " Nov 02 15:25:20.889921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.889797 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-certs\") pod \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " Nov 02 15:25:20.889921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.889836 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-token\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-token\") pod \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " Nov 02 15:25:20.889921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.889874 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-kube-api-access-kz7ch\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-kube-api-access-kz7ch\") pod \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " Nov 02 15:25:20.890158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.889955 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-podinfo\") pod \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " Nov 02 15:25:20.890158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.890031 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-data" pod "4c40bb21-2237-4cb8-8792-9e3dd99d62f7" (UID: "4c40bb21-2237-4cb8-8792-9e3dd99d62f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:20.890158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.890054 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istiod-ca-cert\") pod \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " Nov 02 15:25:20.890158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.890059 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-credential-socket" pod "4c40bb21-2237-4cb8-8792-9e3dd99d62f7" (UID: "4c40bb21-2237-4cb8-8792-9e3dd99d62f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:20.890346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.890188 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-certs" pod "4c40bb21-2237-4cb8-8792-9e3dd99d62f7" (UID: "4c40bb21-2237-4cb8-8792-9e3dd99d62f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:20.890346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.890281 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-socket\") pod \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " Nov 02 15:25:20.890346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.890324 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-envoy\") pod \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\" (UID: \"4c40bb21-2237-4cb8-8792-9e3dd99d62f7\") " Nov 02 15:25:20.891095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.891035 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-socket" pod "4c40bb21-2237-4cb8-8792-9e3dd99d62f7" (UID: "4c40bb21-2237-4cb8-8792-9e3dd99d62f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:20.891276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.891163 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istiod-ca-cert" pod "4c40bb21-2237-4cb8-8792-9e3dd99d62f7" (UID: "4c40bb21-2237-4cb8-8792-9e3dd99d62f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:20.893373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.893343 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:20.893543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.893376 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:20.893543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.893391 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:20.893543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.893406 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:20.893543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.893426 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:20.898209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.898182 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-kube-api-access-kz7ch" pod "4c40bb21-2237-4cb8-8792-9e3dd99d62f7" (UID: "4c40bb21-2237-4cb8-8792-9e3dd99d62f7"). InnerVolumeSpecName "kube-api-access-kz7ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:20.921193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.912729 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-podinfo" pod "4c40bb21-2237-4cb8-8792-9e3dd99d62f7" (UID: "4c40bb21-2237-4cb8-8792-9e3dd99d62f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:25:20.921193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.920550 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-token" pod "4c40bb21-2237-4cb8-8792-9e3dd99d62f7" (UID: "4c40bb21-2237-4cb8-8792-9e3dd99d62f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:20.921925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.921870 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-envoy" pod "4c40bb21-2237-4cb8-8792-9e3dd99d62f7" (UID: "4c40bb21-2237-4cb8-8792-9e3dd99d62f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:20.980578 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.980489 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:20.995078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.995055 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:20.995078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.995076 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:20.995233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.995088 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz7ch\" (UniqueName: \"kubernetes.io/projected/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-kube-api-access-kz7ch\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:20.995233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:20.995100 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c40bb21-2237-4cb8-8792-9e3dd99d62f7-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:21.195180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:21.195138 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:21.418995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:21.418442 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:21.630144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:21.630108 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" event={"ID":"4c40bb21-2237-4cb8-8792-9e3dd99d62f7","Type":"ContainerDied","Data":"266ea17960853404865ddcbbcc0cf9a9667a213b3500429a06c5a6d6ae343801"} Nov 02 15:25:21.630325 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:21.630171 5433 scope.go:122] "RemoveContainer" containerID="5ecbd1337de1755ae7defc865f7780a36895d7ab532bf07d8cabb6ad8f8415ed" Nov 02 15:25:21.630325 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:21.630269 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg" Nov 02 15:25:21.789667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:21.789587 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg"] Nov 02 15:25:21.835881 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:21.835846 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a758f7bf-openshift-default-fcb4ccfd9-rb9xg"] Nov 02 15:25:22.458306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:22.458273 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" path="/var/lib/kubelet/pods/4c40bb21-2237-4cb8-8792-9e3dd99d62f7/volumes" Nov 02 15:25:24.263278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.263244 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:24.555045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.553557 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:24.660612 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.660578 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x"] Nov 02 15:25:24.661391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.661368 5433 cpu_manager.go:414] "containerMap: removing container" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" Nov 02 15:25:24.661568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.661549 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" Nov 02 15:25:24.661692 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.661671 5433 cpu_manager.go:414] "containerMap: removing container" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" Nov 02 15:25:24.661692 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.661686 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" Nov 02 15:25:24.661906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.661893 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" Nov 02 15:25:24.667383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.667364 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:24.721416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.721386 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x"] Nov 02 15:25:24.764483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.764194 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9d0b05fd\"" Nov 02 15:25:24.764483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.764391 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9d0b05fd-dockercfg-s9h68\"" Nov 02 15:25:25.001048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.954080 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dttf\" (UniqueName: \"kubernetes.io/projected/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-kube-api-access-4dttf\") pod \"payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x\" (UID: \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:25.001048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:24.954273 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x\" (UID: \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:25.058655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.058588 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x\" (UID: \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:25.059229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.059200 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dttf\" (UniqueName: \"kubernetes.io/projected/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-kube-api-access-4dttf\") pod \"payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x\" (UID: \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:25.061177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.061154 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x\" (UID: \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:25.061355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.061339 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch"] Nov 02 15:25:25.062783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.062637 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c40bb21-2237-4cb8-8792-9e3dd99d62f7" containerName="istio-proxy" Nov 02 15:25:25.070421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.070296 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:25.181364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.181325 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcfb2\" (UniqueName: \"kubernetes.io/projected/85b21472-eadd-4945-bae1-dc45cdc075c6-kube-api-access-jcfb2\") pod \"payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch\" (UID: \"85b21472-eadd-4945-bae1-dc45cdc075c6\") " pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:25.181672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.181654 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b21472-eadd-4945-bae1-dc45cdc075c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch\" (UID: \"85b21472-eadd-4945-bae1-dc45cdc075c6\") " pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:25.188284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.188251 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch"] Nov 02 15:25:25.192508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.192479 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dttf\" (UniqueName: \"kubernetes.io/projected/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-kube-api-access-4dttf\") pod \"payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x\" (UID: \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:25.281621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.281530 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:25.283309 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.283273 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b21472-eadd-4945-bae1-dc45cdc075c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch\" (UID: \"85b21472-eadd-4945-bae1-dc45cdc075c6\") " pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:25.283450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.283384 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcfb2\" (UniqueName: \"kubernetes.io/projected/85b21472-eadd-4945-bae1-dc45cdc075c6-kube-api-access-jcfb2\") pod \"payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch\" (UID: \"85b21472-eadd-4945-bae1-dc45cdc075c6\") " pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:25.284043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.284017 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b21472-eadd-4945-bae1-dc45cdc075c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch\" (UID: \"85b21472-eadd-4945-bae1-dc45cdc075c6\") " pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:25.323528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.323491 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:25.341604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.341575 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcfb2\" (UniqueName: \"kubernetes.io/projected/85b21472-eadd-4945-bae1-dc45cdc075c6-kube-api-access-jcfb2\") pod \"payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch\" (UID: \"85b21472-eadd-4945-bae1-dc45cdc075c6\") " pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:25.489288 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.489253 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:25.592206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.592168 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5"] Nov 02 15:25:25.599147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.599121 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.628624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.628583 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb"] Nov 02 15:25:25.633444 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.633417 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.689882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.689838 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f450e2-ade2-435a-87d4-fd0cf4693442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.689882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.689877 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.690360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.689986 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgbt\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-kube-api-access-9bgbt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.690360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690058 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.690360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690240 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.690360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690284 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.690577 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690464 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.690577 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690516 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.690680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690590 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.690680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690653 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.690780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690687 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.690830 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690798 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.690830 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690819 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxjk\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-kube-api-access-7sxjk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.690929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690851 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.690929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690879 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.691077 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.690962 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.691143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.691065 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.691196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.691139 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fad2e2e-4765-463c-8062-fa51ddde66db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.694317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.694291 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5"] Nov 02 15:25:25.699352 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:25:25.698381 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54d492f_3f09_4f5e_bed2_9f4394ec9db7.slice/crio-00d62a0e0b528a4fb7692c1bc7911182b3d3b6af0894b2e6bda2098f33d05cb5 WatchSource:0}: Error finding container 00d62a0e0b528a4fb7692c1bc7911182b3d3b6af0894b2e6bda2098f33d05cb5: Status 404 returned error can't find the container with id 00d62a0e0b528a4fb7692c1bc7911182b3d3b6af0894b2e6bda2098f33d05cb5 Nov 02 15:25:25.699352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.699247 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x"] Nov 02 15:25:25.745715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.745686 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb"] Nov 02 15:25:25.793251 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793216 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.793367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793282 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fad2e2e-4765-463c-8062-fa51ddde66db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.793367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793335 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f450e2-ade2-435a-87d4-fd0cf4693442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.793367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793360 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.793525 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793389 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgbt\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-kube-api-access-9bgbt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.793525 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793432 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.793525 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793455 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.793525 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793486 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.793710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793608 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.793710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793653 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.793710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793652 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.793710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793683 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.793863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.793775 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.794039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.794013 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.794169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.794113 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.794274 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.794255 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.794349 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.794313 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.794769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.794424 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.794769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.794673 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.794769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.794707 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.794969 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.794770 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.795406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.795088 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.795406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.795132 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxjk\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-kube-api-access-7sxjk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.795406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.795190 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.795406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.795193 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fad2e2e-4765-463c-8062-fa51ddde66db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.795406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.795128 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f450e2-ade2-435a-87d4-fd0cf4693442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.795406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.795224 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.795406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.795302 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.796552 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.796533 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.797511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.797491 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.797608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.797555 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.797678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.797659 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.852622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.852587 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.852765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.852746 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxjk\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-kube-api-access-7sxjk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4d7h5\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.852808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.852746 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgbt\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-kube-api-access-9bgbt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.852903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.852883 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6hlvb\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:25.865405 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.865383 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch"] Nov 02 15:25:25.865855 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:25:25.865829 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b21472_eadd_4945_bae1_dc45cdc075c6.slice/crio-220cd58f1b7f55048f11bcc00ad65e353d8011c4d6303284353fdf796c488ffb WatchSource:0}: Error finding container 220cd58f1b7f55048f11bcc00ad65e353d8011c4d6303284353fdf796c488ffb: Status 404 returned error can't find the container with id 220cd58f1b7f55048f11bcc00ad65e353d8011c4d6303284353fdf796c488ffb Nov 02 15:25:25.917718 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.917681 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:25.948031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:25.948002 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:26.314710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.314622 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:26.353605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.353017 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5"] Nov 02 15:25:26.353605 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:25:26.353500 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fad2e2e_4765_463c_8062_fa51ddde66db.slice/crio-a32ee5c9e55e9cf65a0937c4bb4fed3d5a69d6c0ea0f68f266b55179729c962a WatchSource:0}: Error finding container a32ee5c9e55e9cf65a0937c4bb4fed3d5a69d6c0ea0f68f266b55179729c962a: Status 404 returned error can't find the container with id a32ee5c9e55e9cf65a0937c4bb4fed3d5a69d6c0ea0f68f266b55179729c962a Nov 02 15:25:26.354791 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:25:26.354761 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f450e2_ade2_435a_87d4_fd0cf4693442.slice/crio-744d0cc9de547b5e8787c734a5a359c295fb76f97e8ae35dd5a0b1c05bfbd386 WatchSource:0}: Error finding container 744d0cc9de547b5e8787c734a5a359c295fb76f97e8ae35dd5a0b1c05bfbd386: Status 404 returned error can't find the container with id 744d0cc9de547b5e8787c734a5a359c295fb76f97e8ae35dd5a0b1c05bfbd386 Nov 02 15:25:26.356738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.356706 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb"] Nov 02 15:25:26.356885 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.356849 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.356950 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.356934 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.357031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.357012 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.357768 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.357731 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.357897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.357813 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.357897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.357857 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.860855 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.860821 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" event={"ID":"3fad2e2e-4765-463c-8062-fa51ddde66db","Type":"ContainerStarted","Data":"a49d3d2105fb431182dcb8c735e13c8f2c62070910b23a20e20511834a85a622"} Nov 02 15:25:26.861153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.861115 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" event={"ID":"3fad2e2e-4765-463c-8062-fa51ddde66db","Type":"ContainerStarted","Data":"a32ee5c9e55e9cf65a0937c4bb4fed3d5a69d6c0ea0f68f266b55179729c962a"} Nov 02 15:25:26.867066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.867033 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" event={"ID":"a54d492f-3f09-4f5e-bed2-9f4394ec9db7","Type":"ContainerStarted","Data":"28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c"} Nov 02 15:25:26.867266 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.867244 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" event={"ID":"a54d492f-3f09-4f5e-bed2-9f4394ec9db7","Type":"ContainerStarted","Data":"00d62a0e0b528a4fb7692c1bc7911182b3d3b6af0894b2e6bda2098f33d05cb5"} Nov 02 15:25:26.868383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.868328 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:26.872419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.872397 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" event={"ID":"85b21472-eadd-4945-bae1-dc45cdc075c6","Type":"ContainerStarted","Data":"49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640"} Nov 02 15:25:26.872604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.872585 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" event={"ID":"85b21472-eadd-4945-bae1-dc45cdc075c6","Type":"ContainerStarted","Data":"220cd58f1b7f55048f11bcc00ad65e353d8011c4d6303284353fdf796c488ffb"} Nov 02 15:25:26.873659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.873641 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:26.885032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.884986 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" event={"ID":"09f450e2-ade2-435a-87d4-fd0cf4693442","Type":"ContainerStarted","Data":"f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c"} Nov 02 15:25:26.885231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.885213 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" event={"ID":"09f450e2-ade2-435a-87d4-fd0cf4693442","Type":"ContainerStarted","Data":"744d0cc9de547b5e8787c734a5a359c295fb76f97e8ae35dd5a0b1c05bfbd386"} Nov 02 15:25:26.926009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.925939 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:26.953137 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:26.951070 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:27.020827 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:27.020753 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podStartSLOduration=2.02073329 podStartE2EDuration="2.02073329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:26.995572451 +0000 UTC m=+1630.668775650" watchObservedRunningTime="2026-11-02 15:25:27.02073329 +0000 UTC m=+1630.693936486" Nov 02 15:25:27.116928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:27.114948 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podStartSLOduration=2.114929047 podStartE2EDuration="2.114929047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:27.112725192 +0000 UTC m=+1630.785928388" watchObservedRunningTime="2026-11-02 15:25:27.114929047 +0000 UTC m=+1630.788132242" Nov 02 15:25:27.323999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:27.292552 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" podStartSLOduration=3.292532183 podStartE2EDuration="3.292532183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:27.224450115 +0000 UTC m=+1630.897653311" watchObservedRunningTime="2026-11-02 15:25:27.292532183 +0000 UTC m=+1630.965735380" Nov 02 15:25:27.409998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:27.391958 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" podStartSLOduration=3.391936085 podStartE2EDuration="3.391936085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:27.362048602 +0000 UTC m=+1631.035251782" watchObservedRunningTime="2026-11-02 15:25:27.391936085 +0000 UTC m=+1631.065139280" Nov 02 15:25:27.950589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:27.950552 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:25:28.006300 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.006246 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 15:25:28.006614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:27.995851 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6hlvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:25:28.006708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.006637 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 15:25:28.082270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.082221 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:28.382264 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.382233 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:28.575100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.568498 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:28.761103 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.750415 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:28.990705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.990667 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6hlvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:25:28.991060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.991006 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 15:25:28.991693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.991671 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:25:28.991826 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:28.991808 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 15:25:29.666390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:29.665704 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:29.858533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:29.858491 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:29.928065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:29.927338 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:25:29.928065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:29.927395 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 15:25:29.953351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:29.953316 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6hlvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:25:29.955878 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:29.955843 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 15:25:30.920403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:30.920364 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:25:30.921529 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:30.921494 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 15:25:30.949374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:30.949337 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6hlvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:25:30.949625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:30.949596 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 15:25:31.477048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:31.476819 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:31.671006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:31.654523 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:31.923998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:31.921399 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:25:31.923998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:31.921472 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 15:25:32.016889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:32.016848 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6hlvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:25:32.017214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:32.017185 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 15:25:32.924789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:32.924367 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:25:32.924789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:32.924429 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 15:25:32.949454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:32.949329 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6hlvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:25:32.949454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:32.949389 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 15:25:33.924002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:33.920371 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:25:33.924002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:33.920440 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 15:25:33.952014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:33.951438 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6hlvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:25:33.952014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:33.951507 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 15:25:34.289518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:34.289421 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:34.928526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:34.928489 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:25:34.928824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:34.928790 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 15:25:34.954840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:34.954807 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6hlvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:25:34.955673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:34.955633 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 15:25:35.918863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:35.918823 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 15:25:35.919176 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:35.919139 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 15:25:35.949316 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:35.949275 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6hlvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:25:35.949498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:35.949337 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 15:25:36.311246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:36.311152 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:36.605210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:36.605113 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:36.771330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:36.771284 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:36.973996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:36.956095 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:37.835933 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:37.835898 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:37.836490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:37.836378 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:37.837344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:37.837296 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:25:37.930330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:37.930296 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4d7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:25:37.930539 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:37.930367 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:25:37.932093 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:37.932061 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:38.066036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:38.066001 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:38.066475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:38.066451 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:38.068533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:38.068508 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:25:38.136825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:38.136779 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:38.285755 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:38.285723 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:38.881178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:38.881145 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:39.224448 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:39.224377 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:25:39.320587 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:39.320555 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:25:41.012538 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:25:41.012498 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:25:42.523010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:42.522897 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:44.855260 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:44.855226 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:48.203169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:48.196070 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s6jxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:25:48.203169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:48.196139 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:25:48.557055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:48.541720 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:48.557055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:48.543751 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:25:48.560998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:48.558057 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:25:48.583998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:48.571286 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:25:48.733512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:48.733465 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:25:48.781282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:48.781238 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:25:53.958577 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:53.958536 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:59.014687 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:59.014647 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:25:59.538081 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:59.538044 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp"] Nov 02 15:25:59.538296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:59.538257 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" containerID="cri-o://accc126df94036061a6f1ba101f39c9aa59fccd641b3a4fc476d7af88df06fbe" gracePeriod=30 Nov 02 15:25:59.582694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:59.582656 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m"] Nov 02 15:25:59.582884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:25:59.582862 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" containerID="cri-o://382bb7dd39f2b8cdefde8ce6928738ee07cf99f2dd78f0af11a9af5f3ef81221" gracePeriod=30 Nov 02 15:26:03.542112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:03.542065 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:04.781150 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:04.780935 5433 generic.go:356] "Generic (PLEG): container finished" podID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerID="382bb7dd39f2b8cdefde8ce6928738ee07cf99f2dd78f0af11a9af5f3ef81221" exitCode=0 Nov 02 15:26:04.781150 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:04.781078 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" event={"ID":"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6","Type":"ContainerDied","Data":"382bb7dd39f2b8cdefde8ce6928738ee07cf99f2dd78f0af11a9af5f3ef81221"} Nov 02 15:26:04.783801 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:04.783652 5433 generic.go:356] "Generic (PLEG): container finished" podID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerID="accc126df94036061a6f1ba101f39c9aa59fccd641b3a4fc476d7af88df06fbe" exitCode=0 Nov 02 15:26:04.783801 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:04.783746 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" event={"ID":"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4","Type":"ContainerDied","Data":"accc126df94036061a6f1ba101f39c9aa59fccd641b3a4fc476d7af88df06fbe"} Nov 02 15:26:04.964621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:04.964589 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:26:05.015667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.015638 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:26:05.066962 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.066932 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-envoy\") pod \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " Nov 02 15:26:05.067147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067131 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istiod-ca-cert\") pod \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " Nov 02 15:26:05.067189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067155 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-credential-socket\") pod \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " Nov 02 15:26:05.067189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067175 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-data\") pod \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " Nov 02 15:26:05.067287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067204 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-certs\") pod \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " Nov 02 15:26:05.067287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067270 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-socket\") pod \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " Nov 02 15:26:05.067425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067405 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-podinfo\") pod \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " Nov 02 15:26:05.067650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067512 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-data" pod "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" (UID: "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.067741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067604 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-certs" pod "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" (UID: "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.067741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067664 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-token\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-token\") pod \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " Nov 02 15:26:05.067741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067707 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-kube-api-access-tn7nv\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-kube-api-access-tn7nv\") pod \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\" (UID: \"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4\") " Nov 02 15:26:05.067741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067619 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-socket" pod "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" (UID: "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.067959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067683 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istiod-ca-cert" pod "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" (UID: "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:26:05.067959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.067718 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-credential-socket" pod "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" (UID: "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.068646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.068623 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.068714 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.068655 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.068714 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.068672 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.068714 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.068686 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.068714 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.068699 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.069331 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.069310 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-envoy" pod "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" (UID: "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.069758 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.069738 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-token" pod "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" (UID: "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:05.070006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.069985 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-podinfo" pod "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" (UID: "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:26:05.070101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.070082 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-kube-api-access-tn7nv" pod "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" (UID: "0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4"). InnerVolumeSpecName "kube-api-access-tn7nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:05.169392 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169354 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-socket\") pod \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " Nov 02 15:26:05.169589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169409 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-token\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-token\") pod \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " Nov 02 15:26:05.169589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169551 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-podinfo\") pod \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " Nov 02 15:26:05.169715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169603 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-certs\") pod \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " Nov 02 15:26:05.169715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169669 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-socket" pod "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" (UID: "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.169715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169700 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istiod-ca-cert\") pod \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " Nov 02 15:26:05.169924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169760 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-envoy\") pod \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " Nov 02 15:26:05.169924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169841 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-kube-api-access-jpjl4\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-kube-api-access-jpjl4\") pod \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " Nov 02 15:26:05.169924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169910 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-certs" pod "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" (UID: "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.170109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169939 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-data\") pod \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " Nov 02 15:26:05.170109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.169963 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-credential-socket\") pod \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\" (UID: \"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6\") " Nov 02 15:26:05.170222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.170168 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-data" pod "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" (UID: "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.170402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.170188 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istiod-ca-cert" pod "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" (UID: "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:26:05.170526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.170506 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-credential-socket" pod "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" (UID: "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.171844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.170873 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn7nv\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-kube-api-access-tn7nv\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.171844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.170901 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.171844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.170917 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.171844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.170931 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.171844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.170947 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.171844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.170961 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.171844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.171038 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.171844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.171054 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.171844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.171070 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.172395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.172362 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-podinfo" pod "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" (UID: "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:26:05.172524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.172494 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-token" pod "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" (UID: "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:05.172697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.172675 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-kube-api-access-jpjl4" pod "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" (UID: "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6"). InnerVolumeSpecName "kube-api-access-jpjl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:05.172766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.172743 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-envoy" pod "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" (UID: "a51d10a9-c5d2-4026-ad91-7addeb1f1fa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.210177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.210123 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:05.275907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.275870 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.275907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.275909 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpjl4\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-kube-api-access-jpjl4\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.276135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.275927 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.276135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.275940 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.426134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.426045 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:05.606010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.600920 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:05.796728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.796626 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" Nov 02 15:26:05.796728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.796652 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m" event={"ID":"a51d10a9-c5d2-4026-ad91-7addeb1f1fa6","Type":"ContainerDied","Data":"c3ce08e3008e619fc729c943b5310390a1fd1043344943f1e0b67066f82e136e"} Nov 02 15:26:05.797287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.796731 5433 scope.go:122] "RemoveContainer" containerID="382bb7dd39f2b8cdefde8ce6928738ee07cf99f2dd78f0af11a9af5f3ef81221" Nov 02 15:26:05.798722 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.798700 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" Nov 02 15:26:05.798818 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.798747 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp" event={"ID":"0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4","Type":"ContainerDied","Data":"07c9fc05618e3be896233de7aa2161bb230fff4aafdf431be9b4c8667d4c449c"} Nov 02 15:26:05.810429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.810409 5433 scope.go:122] "RemoveContainer" containerID="accc126df94036061a6f1ba101f39c9aa59fccd641b3a4fc476d7af88df06fbe" Nov 02 15:26:05.989997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:05.983311 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m"] Nov 02 15:26:06.048049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:06.042229 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-h8p4m"] Nov 02 15:26:06.106043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:06.104088 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp"] Nov 02 15:26:06.132000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:06.120859 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-5wbvp"] Nov 02 15:26:06.371485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:06.371410 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:06.533320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:06.533272 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" path="/var/lib/kubelet/pods/0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4/volumes" Nov 02 15:26:06.556006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:06.554671 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" path="/var/lib/kubelet/pods/a51d10a9-c5d2-4026-ad91-7addeb1f1fa6/volumes" Nov 02 15:26:07.258373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:07.258343 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:07.407115 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:07.407079 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:07.703303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:07.703269 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:09.277104 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:09.277072 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:10.658636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.658599 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4"] Nov 02 15:26:10.659560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.659536 5433 cpu_manager.go:414] "containerMap: removing container" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" Nov 02 15:26:10.659560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.659560 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" Nov 02 15:26:10.659729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.659573 5433 cpu_manager.go:414] "containerMap: removing container" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" Nov 02 15:26:10.659729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.659580 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" Nov 02 15:26:10.659825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.659758 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0280aa67-0b9a-4ae7-a3b6-2efc3b875fc4" containerName="istio-proxy" Nov 02 15:26:10.659825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.659777 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a51d10a9-c5d2-4026-ad91-7addeb1f1fa6" containerName="istio-proxy" Nov 02 15:26:10.665181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.665156 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.761804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.761766 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9d0b05fd-serving-cert\"" Nov 02 15:26:10.768311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.768275 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4"] Nov 02 15:26:10.823528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.823495 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.823694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.823562 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4wxj\" (UniqueName: \"kubernetes.io/projected/c00e11f9-6483-450f-ab63-8f35b4187944-kube-api-access-l4wxj\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.823780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.823755 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-metrics-certs\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.824047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.824023 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00e11f9-6483-450f-ab63-8f35b4187944-openshift-service-ca\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.865066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.865028 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:10.926008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.925876 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00e11f9-6483-450f-ab63-8f35b4187944-openshift-service-ca\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.926181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.926086 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.926181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.926146 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4wxj\" (UniqueName: \"kubernetes.io/projected/c00e11f9-6483-450f-ab63-8f35b4187944-kube-api-access-l4wxj\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.926301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.926183 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-metrics-certs\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.926301 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:26:10.926235 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-serving-cert: secret "maas-api-e2e-ait-9d0b05fd-serving-cert" not found Nov 02 15:26:10.926404 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:26:10.926321 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls podName:c00e11f9-6483-450f-ab63-8f35b4187944 nodeName:}" failed. No retries permitted until 2026-11-02 15:26:11.426299728 +0000 UTC m=+1675.099502902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls") pod "maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" (UID: "c00e11f9-6483-450f-ab63-8f35b4187944") : secret "maas-api-e2e-ait-9d0b05fd-serving-cert" not found Nov 02 15:26:10.926561 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.926539 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00e11f9-6483-450f-ab63-8f35b4187944-openshift-service-ca\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.928814 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.928790 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-metrics-certs\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:10.988478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:10.988450 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4wxj\" (UniqueName: \"kubernetes.io/projected/c00e11f9-6483-450f-ab63-8f35b4187944-kube-api-access-l4wxj\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:11.016675 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:26:11.016636 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:26:11.028793 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:11.028763 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:11.181096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:11.178146 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:11.441416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:11.441336 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:11.443770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:11.443740 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls\") pod \"maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:11.580158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:11.580126 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:11.955058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:11.955028 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4"] Nov 02 15:26:11.956373 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:26:11.956345 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00e11f9_6483_450f_ab63_8f35b4187944.slice/crio-617460217d75d9d5a959687d161e941cd6cc1923e4bc5e564113f9e7dfa246eb WatchSource:0}: Error finding container 617460217d75d9d5a959687d161e941cd6cc1923e4bc5e564113f9e7dfa246eb: Status 404 returned error can't find the container with id 617460217d75d9d5a959687d161e941cd6cc1923e4bc5e564113f9e7dfa246eb Nov 02 15:26:11.998732 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:11.998695 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:12.981958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:12.981923 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" event={"ID":"c00e11f9-6483-450f-ab63-8f35b4187944","Type":"ContainerStarted","Data":"617460217d75d9d5a959687d161e941cd6cc1923e4bc5e564113f9e7dfa246eb"} Nov 02 15:26:14.040680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:14.040573 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:14.043625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:14.043586 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" event={"ID":"c00e11f9-6483-450f-ab63-8f35b4187944","Type":"ContainerStarted","Data":"5b128d98b929222931fd4b76f61e18a8de94b9fd7800736cd34e00a77dc025a8"} Nov 02 15:26:14.044071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:14.044053 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:14.172607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:14.172551 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" podStartSLOduration=3.158937575 podStartE2EDuration="4.172536498s" totalImagesPullingTime="1.013598923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:14.168111052 +0000 UTC m=+1677.841314244" watchObservedRunningTime="2026-11-02 15:26:14.172536498 +0000 UTC m=+1677.845739693" Nov 02 15:26:14.209384 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:14.209350 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:15.843732 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:15.843698 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:16.101873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:16.101785 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:16.488072 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:16.479961 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:16.831061 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:16.822759 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:19.119757 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:19.119721 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:19.856614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:19.856534 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:20.068774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:20.068749 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:26:20.590116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:20.588579 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:20.802515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:20.802481 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:20.970512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:20.970477 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:21.116335 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:21.116304 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:21.702207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:21.702162 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:24.010736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:24.010701 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:24.192464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:24.192425 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:25.682391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:25.682353 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:26.758776 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:26.758740 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:27.713185 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:27.713146 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:28.985306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:28.985242 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:29.210615 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.210545 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:29.676588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.676557 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz"] Nov 02 15:26:29.703234 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.703197 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz"] Nov 02 15:26:29.704055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.704031 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:29.793998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.791241 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz\" (UID: \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:29.793998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.791413 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnxb\" (UniqueName: \"kubernetes.io/projected/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-kube-api-access-fwnxb\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz\" (UID: \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:29.844999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.841284 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5aeac88e\"" Nov 02 15:26:29.844999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.841400 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5aeac88e-dockercfg-m4wwh\"" Nov 02 15:26:29.896112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.896075 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz\" (UID: \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:29.915315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.915276 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnxb\" (UniqueName: \"kubernetes.io/projected/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-kube-api-access-fwnxb\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz\" (UID: \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:29.916240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:29.897114 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz\" (UID: \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:30.030048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:30.025796 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnxb\" (UniqueName: \"kubernetes.io/projected/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-kube-api-access-fwnxb\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz\" (UID: \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:30.192506 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:30.192472 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:30.565480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:30.565450 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz"] Nov 02 15:26:30.566806 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:26:30.566777 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b66d9a_4ea2_4d8f_aeee_e7336428164e.slice/crio-d192901897869e0fffe622fac91306f72c88d3cb33b95e7148a3e413b8eeefe2 WatchSource:0}: Error finding container d192901897869e0fffe622fac91306f72c88d3cb33b95e7148a3e413b8eeefe2: Status 404 returned error can't find the container with id d192901897869e0fffe622fac91306f72c88d3cb33b95e7148a3e413b8eeefe2 Nov 02 15:26:30.676959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:30.676925 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" event={"ID":"c1b66d9a-4ea2-4d8f-aeee-e7336428164e","Type":"ContainerStarted","Data":"8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f"} Nov 02 15:26:30.677147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:30.677025 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" event={"ID":"c1b66d9a-4ea2-4d8f-aeee-e7336428164e","Type":"ContainerStarted","Data":"d192901897869e0fffe622fac91306f72c88d3cb33b95e7148a3e413b8eeefe2"} Nov 02 15:26:30.677147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:30.677055 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:30.789916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:30.789860 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" podStartSLOduration=1.789847087 podStartE2EDuration="1.789847087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:30.786045105 +0000 UTC m=+1694.459248313" watchObservedRunningTime="2026-11-02 15:26:30.789847087 +0000 UTC m=+1694.463050299" Nov 02 15:26:32.565268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:32.565165 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:33.683203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:33.683162 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:33.834813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:33.834761 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:33.994902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:33.994612 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:37.663776 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:37.663694 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:37.849732 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:37.849702 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:38.042214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:38.042119 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:39.482156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:39.482125 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:40.942137 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:40.942093 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:41.012317 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:26:41.012281 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:26:41.313056 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:41.308579 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:41.768005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:41.761181 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:26:45.162318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:45.162281 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:45.993034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:45.992949 5433 generic.go:356] "Generic (PLEG): container finished" podID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerID="066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce" exitCode=0 Nov 02 15:26:45.993034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:45.993009 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" event={"ID":"c8622a11-4189-477c-bd43-1cc6d12250ca","Type":"ContainerDied","Data":"066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce"} Nov 02 15:26:45.993447 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:45.993432 5433 scope.go:122] "RemoveContainer" containerID="066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce" Nov 02 15:26:45.995155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:45.995113 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:45.995311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:45.995241 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:45.995311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:45.995303 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:47.001762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:47.001728 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" event={"ID":"c8622a11-4189-477c-bd43-1cc6d12250ca","Type":"ContainerStarted","Data":"31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9"} Nov 02 15:26:47.710789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:47.710754 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:26:47.711764 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:47.711732 5433 patch_prober.go:29] interesting pod/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:26:47.711842 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:47.711796 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 15:26:48.710951 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:48.710913 5433 patch_prober.go:29] interesting pod/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:26:48.711459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:48.710970 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 15:26:49.711301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:49.711255 5433 patch_prober.go:29] interesting pod/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:26:49.711696 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:49.711337 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 15:26:50.541002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:50.538577 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:26:50.637005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:50.622046 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:26:50.712006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:50.711024 5433 patch_prober.go:29] interesting pod/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:26:50.712006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:50.711113 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 15:26:51.711351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:51.711315 5433 patch_prober.go:29] interesting pod/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:26:51.711522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:51.711387 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 15:26:52.716404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:52.716368 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:26:52.717420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:52.717398 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:26:55.667109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:26:55.667072 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:00.771312 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:00.771225 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:06.157835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:06.157788 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:11.015368 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:11.015317 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:27:11.219947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.219912 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4"] Nov 02 15:27:11.224742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.224716 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.318845 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.318753 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-metrics-certs\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.319006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.318883 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d117b852-c584-484a-8b70-a5a87f32a82d-openshift-service-ca\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.319054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.319022 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.319089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.319070 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntmq\" (UniqueName: \"kubernetes.io/projected/d117b852-c584-484a-8b70-a5a87f32a82d-kube-api-access-zntmq\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.323899 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.323870 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4"] Nov 02 15:27:11.332552 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.332528 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d5b52693-serving-cert\"" Nov 02 15:27:11.420917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.420878 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d117b852-c584-484a-8b70-a5a87f32a82d-openshift-service-ca\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.421116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.421012 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.421116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.421039 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntmq\" (UniqueName: \"kubernetes.io/projected/d117b852-c584-484a-8b70-a5a87f32a82d-kube-api-access-zntmq\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.421213 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:11.421148 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-serving-cert: secret "maas-api-e2e-mt-d5b52693-serving-cert" not found Nov 02 15:27:11.421253 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:11.421221 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls podName:d117b852-c584-484a-8b70-a5a87f32a82d nodeName:}" failed. No retries permitted until 2026-11-02 15:27:11.921198909 +0000 UTC m=+1735.594402082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls") pod "maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" (UID: "d117b852-c584-484a-8b70-a5a87f32a82d") : secret "maas-api-e2e-mt-d5b52693-serving-cert" not found Nov 02 15:27:11.421309 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.421290 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-metrics-certs\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.421681 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.421663 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d117b852-c584-484a-8b70-a5a87f32a82d-openshift-service-ca\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.423920 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.423890 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-metrics-certs\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.424062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.423961 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:11.526386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.526351 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntmq\" (UniqueName: \"kubernetes.io/projected/d117b852-c584-484a-8b70-a5a87f32a82d-kube-api-access-zntmq\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.671999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.665029 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:11.803894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.803813 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:11.932751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.932671 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:11.935156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:11.935133 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls\") pod \"maas-api-e2e-mt-d5b52693-78896d4696-2mfm4\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:12.140340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:12.140305 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:12.520922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:12.520882 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4"] Nov 02 15:27:12.527946 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:27:12.527911 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd117b852_c584_484a_8b70_a5a87f32a82d.slice/crio-32aa577636861ad77f125be5ee5f5d95261627826aa6e3eb38ba85f8c882a0da WatchSource:0}: Error finding container 32aa577636861ad77f125be5ee5f5d95261627826aa6e3eb38ba85f8c882a0da: Status 404 returned error can't find the container with id 32aa577636861ad77f125be5ee5f5d95261627826aa6e3eb38ba85f8c882a0da Nov 02 15:27:13.376308 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:13.376267 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" event={"ID":"d117b852-c584-484a-8b70-a5a87f32a82d","Type":"ContainerStarted","Data":"32aa577636861ad77f125be5ee5f5d95261627826aa6e3eb38ba85f8c882a0da"} Nov 02 15:27:13.783245 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:13.783207 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:14.385332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:14.385290 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" event={"ID":"d117b852-c584-484a-8b70-a5a87f32a82d","Type":"ContainerStarted","Data":"6d279aa6174a3485479c4264c8001a497608dbd70633b4d3b8b13312cf2a3938"} Nov 02 15:27:14.385806 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:14.385357 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:14.498444 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:14.498387 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" podStartSLOduration=2.440132861 podStartE2EDuration="3.498374321s" totalImagesPullingTime="1.05824146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:14.495852734 +0000 UTC m=+1738.169055930" watchObservedRunningTime="2026-11-02 15:27:14.498374321 +0000 UTC m=+1738.171577515" Nov 02 15:27:16.999646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:16.999610 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j"] Nov 02 15:27:17.000083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:16.999806 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" containerID="cri-o://35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3" gracePeriod=30 Nov 02 15:27:17.046342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:17.046300 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc"] Nov 02 15:27:17.046534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:17.046513 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" containerID="cri-o://db34fca964b0aac202e2f6ccb8dfc5a5c2c018e00dac826fff17cd951fd66b17" gracePeriod=30 Nov 02 15:27:17.049943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:17.049911 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh"] Nov 02 15:27:17.050176 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:17.050147 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" containerID="cri-o://054d4d6a9dbe8bd7b7e9a5cbd17da346e3a0b9b7316357d0e4d674b996aa7eaa" gracePeriod=30 Nov 02 15:27:17.092085 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:17.092049 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc"] Nov 02 15:27:17.092330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:17.092295 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" containerID="cri-o://99958702c4af86ad6a260f9483094fdc0358338baf8429814b112f218bb879b8" gracePeriod=30 Nov 02 15:27:17.099364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:17.099311 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:20.398205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:20.398172 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:27:20.428196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:20.428163 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:22.204132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.204102 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb"] Nov 02 15:27:22.209423 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.209401 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.298739 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.296965 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5aeac88e-serving-cert\"" Nov 02 15:27:22.302134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.301230 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb"] Nov 02 15:27:22.346055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.345954 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e95535-82c8-4251-80d3-da54d4f1c6d2-openshift-service-ca\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.346215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.346171 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-259hl\" (UniqueName: \"kubernetes.io/projected/95e95535-82c8-4251-80d3-da54d4f1c6d2-kube-api-access-259hl\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.346318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.346287 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.346559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.346532 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-metrics-certs\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.366502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.366478 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:27:22.410240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.410202 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:22.448160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448122 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-credential-socket\") pod \"26cf8c8a-484a-42d7-b251-b4950a0c052a\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " Nov 02 15:27:22.448323 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448221 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-kube-api-access-5sxj7\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-kube-api-access-5sxj7\") pod \"26cf8c8a-484a-42d7-b251-b4950a0c052a\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " Nov 02 15:27:22.448400 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448366 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cf8c8a-484a-42d7-b251-b4950a0c052a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26cf8c8a-484a-42d7-b251-b4950a0c052a-istiod-ca-cert\") pod \"26cf8c8a-484a-42d7-b251-b4950a0c052a\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " Nov 02 15:27:22.448458 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448400 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-certs\") pod \"26cf8c8a-484a-42d7-b251-b4950a0c052a\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " Nov 02 15:27:22.448458 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448432 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-data\") pod \"26cf8c8a-484a-42d7-b251-b4950a0c052a\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " Nov 02 15:27:22.448571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448464 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-token\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-token\") pod \"26cf8c8a-484a-42d7-b251-b4950a0c052a\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " Nov 02 15:27:22.448626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448593 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-socket\") pod \"26cf8c8a-484a-42d7-b251-b4950a0c052a\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " Nov 02 15:27:22.448626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448621 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-envoy\") pod \"26cf8c8a-484a-42d7-b251-b4950a0c052a\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " Nov 02 15:27:22.448730 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448655 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-podinfo\") pod \"26cf8c8a-484a-42d7-b251-b4950a0c052a\" (UID: \"26cf8c8a-484a-42d7-b251-b4950a0c052a\") " Nov 02 15:27:22.450143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.448965 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.450143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.449160 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-metrics-certs\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.450143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.449363 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e95535-82c8-4251-80d3-da54d4f1c6d2-openshift-service-ca\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.450143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.449509 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259hl\" (UniqueName: \"kubernetes.io/projected/95e95535-82c8-4251-80d3-da54d4f1c6d2-kube-api-access-259hl\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.450511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.450478 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-credential-socket" pod "26cf8c8a-484a-42d7-b251-b4950a0c052a" (UID: "26cf8c8a-484a-42d7-b251-b4950a0c052a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.452775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.452748 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-socket" pod "26cf8c8a-484a-42d7-b251-b4950a0c052a" (UID: "26cf8c8a-484a-42d7-b251-b4950a0c052a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.453081 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.453053 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cf8c8a-484a-42d7-b251-b4950a0c052a-istiod-ca-cert" pod "26cf8c8a-484a-42d7-b251-b4950a0c052a" (UID: "26cf8c8a-484a-42d7-b251-b4950a0c052a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:22.453298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.453278 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-certs" pod "26cf8c8a-484a-42d7-b251-b4950a0c052a" (UID: "26cf8c8a-484a-42d7-b251-b4950a0c052a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.453377 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.453347 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-kube-api-access-5sxj7" pod "26cf8c8a-484a-42d7-b251-b4950a0c052a" (UID: "26cf8c8a-484a-42d7-b251-b4950a0c052a"). InnerVolumeSpecName "kube-api-access-5sxj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.453466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.453449 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-data" pod "26cf8c8a-484a-42d7-b251-b4950a0c052a" (UID: "26cf8c8a-484a-42d7-b251-b4950a0c052a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.456175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.456108 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e95535-82c8-4251-80d3-da54d4f1c6d2-openshift-service-ca\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.457286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.457154 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-envoy" pod "26cf8c8a-484a-42d7-b251-b4950a0c052a" (UID: "26cf8c8a-484a-42d7-b251-b4950a0c052a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.459401 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.459151 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-podinfo" pod "26cf8c8a-484a-42d7-b251-b4950a0c052a" (UID: "26cf8c8a-484a-42d7-b251-b4950a0c052a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:22.460015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.459991 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-metrics-certs\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.464368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.464346 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-token" pod "26cf8c8a-484a-42d7-b251-b4950a0c052a" (UID: "26cf8c8a-484a-42d7-b251-b4950a0c052a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.464475 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:22.464461 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-serving-cert: secret "maas-api-e2e-mt-5aeac88e-serving-cert" not found Nov 02 15:27:22.464593 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:22.464524 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls podName:95e95535-82c8-4251-80d3-da54d4f1c6d2 nodeName:}" failed. No retries permitted until 2026-11-02 15:27:22.964503728 +0000 UTC m=+1746.637706907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls") pod "maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" (UID: "95e95535-82c8-4251-80d3-da54d4f1c6d2") : secret "maas-api-e2e-mt-5aeac88e-serving-cert" not found Nov 02 15:27:22.465431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.465406 5433 generic.go:356] "Generic (PLEG): container finished" podID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerID="db34fca964b0aac202e2f6ccb8dfc5a5c2c018e00dac826fff17cd951fd66b17" exitCode=0 Nov 02 15:27:22.467731 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.467696 5433 generic.go:356] "Generic (PLEG): container finished" podID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerID="054d4d6a9dbe8bd7b7e9a5cbd17da346e3a0b9b7316357d0e4d674b996aa7eaa" exitCode=0 Nov 02 15:27:22.473028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.472577 5433 generic.go:356] "Generic (PLEG): container finished" podID="d4df3520-364b-4282-a066-a83c7e2ff592" containerID="99958702c4af86ad6a260f9483094fdc0358338baf8429814b112f218bb879b8" exitCode=0 Nov 02 15:27:22.476533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.475722 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" event={"ID":"d6f141fd-aa62-4bb8-90db-c5ee9518b194","Type":"ContainerDied","Data":"db34fca964b0aac202e2f6ccb8dfc5a5c2c018e00dac826fff17cd951fd66b17"} Nov 02 15:27:22.476533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.475796 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" event={"ID":"f593f01e-a1ad-4ebd-b605-af6e203a8b7e","Type":"ContainerDied","Data":"054d4d6a9dbe8bd7b7e9a5cbd17da346e3a0b9b7316357d0e4d674b996aa7eaa"} Nov 02 15:27:22.476533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.475835 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" event={"ID":"d4df3520-364b-4282-a066-a83c7e2ff592","Type":"ContainerDied","Data":"99958702c4af86ad6a260f9483094fdc0358338baf8429814b112f218bb879b8"} Nov 02 15:27:22.476533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.475888 5433 scope.go:122] "RemoveContainer" containerID="903ac58cc6aabd07075250b8963819a3cd59fa62db60cc4b9aa64b20696d74b4" Nov 02 15:27:22.496707 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.496680 5433 generic.go:356] "Generic (PLEG): container finished" podID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerID="35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3" exitCode=0 Nov 02 15:27:22.496911 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.496891 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" event={"ID":"26cf8c8a-484a-42d7-b251-b4950a0c052a","Type":"ContainerDied","Data":"35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3"} Nov 02 15:27:22.497104 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.497078 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" event={"ID":"26cf8c8a-484a-42d7-b251-b4950a0c052a","Type":"ContainerDied","Data":"55acb86690ac1efc5dab31a274b07bdf3ec8ace95ccad559bb731b653279d616"} Nov 02 15:27:22.497412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.497393 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j" Nov 02 15:27:22.513055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.513028 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-259hl\" (UniqueName: \"kubernetes.io/projected/95e95535-82c8-4251-80d3-da54d4f1c6d2-kube-api-access-259hl\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.516058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.516033 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:27:22.523865 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.523848 5433 scope.go:122] "RemoveContainer" containerID="35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3" Nov 02 15:27:22.537342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.537313 5433 scope.go:122] "RemoveContainer" containerID="35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3" Nov 02 15:27:22.537342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.537339 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:27:22.537666 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:22.537636 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3\": container with ID starting with 35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3 not found: ID does not exist" containerID="35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3" Nov 02 15:27:22.537831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.537798 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3"} err="failed to get container status \"35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3\": rpc error: code = NotFound desc = could not find container \"35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3\": container with ID starting with 35d930bd72a6140cd1130bd51390901f4b326b74232ddbef834d7e3c314cf2b3 not found: ID does not exist" Nov 02 15:27:22.541858 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.541835 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:27:22.552139 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.552113 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxj7\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-kube-api-access-5sxj7\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.552139 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.552139 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26cf8c8a-484a-42d7-b251-b4950a0c052a-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.552303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.552154 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.552303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.552163 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.552303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.552171 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.552303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.552180 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.552303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.552188 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.552303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.552196 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26cf8c8a-484a-42d7-b251-b4950a0c052a-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.552303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.552204 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26cf8c8a-484a-42d7-b251-b4950a0c052a-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.577101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.577075 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:22.672389 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.656515 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-credential-socket\") pod \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " Nov 02 15:27:22.672389 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.656583 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-socket\") pod \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " Nov 02 15:27:22.672389 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.656620 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-envoy\") pod \"d4df3520-364b-4282-a066-a83c7e2ff592\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " Nov 02 15:27:22.674856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.674822 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-certs\") pod \"d4df3520-364b-4282-a066-a83c7e2ff592\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " Nov 02 15:27:22.675143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.675113 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-podinfo\") pod \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " Nov 02 15:27:22.687805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.675388 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-envoy\") pod \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " Nov 02 15:27:22.689158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.689129 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-data\") pod \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " Nov 02 15:27:22.691095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.691069 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-socket\") pod \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " Nov 02 15:27:22.691311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.691294 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4df3520-364b-4282-a066-a83c7e2ff592-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4df3520-364b-4282-a066-a83c7e2ff592-istio-podinfo\") pod \"d4df3520-364b-4282-a066-a83c7e2ff592\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " Nov 02 15:27:22.691459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.691443 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-envoy\") pod \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " Nov 02 15:27:22.691587 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.691574 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-data\") pod \"d4df3520-364b-4282-a066-a83c7e2ff592\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " Nov 02 15:27:22.691701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.691684 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istiod-ca-cert\") pod \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " Nov 02 15:27:22.691880 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.691864 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-certs\") pod \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " Nov 02 15:27:22.692018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.692003 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-token\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-token\") pod \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " Nov 02 15:27:22.712000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.692153 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istiod-ca-cert\") pod \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " Nov 02 15:27:22.712284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.712258 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-credential-socket\") pod \"d4df3520-364b-4282-a066-a83c7e2ff592\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " Nov 02 15:27:22.712483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.712464 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-podinfo\") pod \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " Nov 02 15:27:22.712601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.712582 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-kube-api-access-sf8xn\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-kube-api-access-sf8xn\") pod \"d4df3520-364b-4282-a066-a83c7e2ff592\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " Nov 02 15:27:22.712736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.712720 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-credential-socket\") pod \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " Nov 02 15:27:22.712919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.712903 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-kube-api-access-84888\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-kube-api-access-84888\") pod \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " Nov 02 15:27:22.713059 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.713042 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4df3520-364b-4282-a066-a83c7e2ff592-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4df3520-364b-4282-a066-a83c7e2ff592-istiod-ca-cert\") pod \"d4df3520-364b-4282-a066-a83c7e2ff592\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " Nov 02 15:27:22.713296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.713277 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-kube-api-access-hknx8\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-kube-api-access-hknx8\") pod \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\" (UID: \"d6f141fd-aa62-4bb8-90db-c5ee9518b194\") " Nov 02 15:27:22.713485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.713469 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-socket\") pod \"d4df3520-364b-4282-a066-a83c7e2ff592\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " Nov 02 15:27:22.713630 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.713616 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-certs\") pod \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " Nov 02 15:27:22.713772 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.713757 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-token\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-token\") pod \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " Nov 02 15:27:22.713902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.713886 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-istio-token\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-istio-token\") pod \"d4df3520-364b-4282-a066-a83c7e2ff592\" (UID: \"d4df3520-364b-4282-a066-a83c7e2ff592\") " Nov 02 15:27:22.714059 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.714044 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-data\") pod \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\" (UID: \"f593f01e-a1ad-4ebd-b605-af6e203a8b7e\") " Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.688308 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-certs" pod "d4df3520-364b-4282-a066-a83c7e2ff592" (UID: "d4df3520-364b-4282-a066-a83c7e2ff592"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.688510 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-credential-socket" pod "f593f01e-a1ad-4ebd-b605-af6e203a8b7e" (UID: "f593f01e-a1ad-4ebd-b605-af6e203a8b7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.689032 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-socket" pod "d6f141fd-aa62-4bb8-90db-c5ee9518b194" (UID: "d6f141fd-aa62-4bb8-90db-c5ee9518b194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.708257 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-data" pod "d6f141fd-aa62-4bb8-90db-c5ee9518b194" (UID: "d6f141fd-aa62-4bb8-90db-c5ee9518b194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.708449 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-socket" pod "f593f01e-a1ad-4ebd-b605-af6e203a8b7e" (UID: "f593f01e-a1ad-4ebd-b605-af6e203a8b7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.708823 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istiod-ca-cert" pod "d6f141fd-aa62-4bb8-90db-c5ee9518b194" (UID: "d6f141fd-aa62-4bb8-90db-c5ee9518b194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.709210 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-data" pod "d4df3520-364b-4282-a066-a83c7e2ff592" (UID: "d4df3520-364b-4282-a066-a83c7e2ff592"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.709596 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-certs" pod "d6f141fd-aa62-4bb8-90db-c5ee9518b194" (UID: "d6f141fd-aa62-4bb8-90db-c5ee9518b194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.725201 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istiod-ca-cert" pod "f593f01e-a1ad-4ebd-b605-af6e203a8b7e" (UID: "f593f01e-a1ad-4ebd-b605-af6e203a8b7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.726099 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-credential-socket" pod "d6f141fd-aa62-4bb8-90db-c5ee9518b194" (UID: "d6f141fd-aa62-4bb8-90db-c5ee9518b194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.726318 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-socket" pod "d4df3520-364b-4282-a066-a83c7e2ff592" (UID: "d4df3520-364b-4282-a066-a83c7e2ff592"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.727058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.726638 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4df3520-364b-4282-a066-a83c7e2ff592-istiod-ca-cert" pod "d4df3520-364b-4282-a066-a83c7e2ff592" (UID: "d4df3520-364b-4282-a066-a83c7e2ff592"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:22.731948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.729153 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-credential-socket" pod "d4df3520-364b-4282-a066-a83c7e2ff592" (UID: "d4df3520-364b-4282-a066-a83c7e2ff592"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.737157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.732668 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-certs" pod "f593f01e-a1ad-4ebd-b605-af6e203a8b7e" (UID: "f593f01e-a1ad-4ebd-b605-af6e203a8b7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.748502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.741216 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-envoy" pod "f593f01e-a1ad-4ebd-b605-af6e203a8b7e" (UID: "f593f01e-a1ad-4ebd-b605-af6e203a8b7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.748502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.742859 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-token" pod "d6f141fd-aa62-4bb8-90db-c5ee9518b194" (UID: "d6f141fd-aa62-4bb8-90db-c5ee9518b194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.748502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.748300 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-data" pod "f593f01e-a1ad-4ebd-b605-af6e203a8b7e" (UID: "f593f01e-a1ad-4ebd-b605-af6e203a8b7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.764797 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-token" pod "f593f01e-a1ad-4ebd-b605-af6e203a8b7e" (UID: "f593f01e-a1ad-4ebd-b605-af6e203a8b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765563 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765592 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765606 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765620 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765634 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765647 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765664 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765681 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765694 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765710 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765724 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765737 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765751 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4df3520-364b-4282-a066-a83c7e2ff592-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765765 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765778 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765791 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765804 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.765817 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.775038 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-podinfo" pod "f593f01e-a1ad-4ebd-b605-af6e203a8b7e" (UID: "f593f01e-a1ad-4ebd-b605-af6e203a8b7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.775188 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4df3520-364b-4282-a066-a83c7e2ff592-istio-podinfo" pod "d4df3520-364b-4282-a066-a83c7e2ff592" (UID: "d4df3520-364b-4282-a066-a83c7e2ff592"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:22.776783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.775384 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-kube-api-access-sf8xn" pod "d4df3520-364b-4282-a066-a83c7e2ff592" (UID: "d4df3520-364b-4282-a066-a83c7e2ff592"). InnerVolumeSpecName "kube-api-access-sf8xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.791903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.778476 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-kube-api-access-hknx8" pod "d6f141fd-aa62-4bb8-90db-c5ee9518b194" (UID: "d6f141fd-aa62-4bb8-90db-c5ee9518b194"). InnerVolumeSpecName "kube-api-access-hknx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.791903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.778562 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-istio-token" pod "d4df3520-364b-4282-a066-a83c7e2ff592" (UID: "d4df3520-364b-4282-a066-a83c7e2ff592"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.791903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.780179 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-podinfo" pod "d6f141fd-aa62-4bb8-90db-c5ee9518b194" (UID: "d6f141fd-aa62-4bb8-90db-c5ee9518b194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:22.791903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.780253 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-envoy" pod "d4df3520-364b-4282-a066-a83c7e2ff592" (UID: "d4df3520-364b-4282-a066-a83c7e2ff592"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.791903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.780304 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-envoy" pod "d6f141fd-aa62-4bb8-90db-c5ee9518b194" (UID: "d6f141fd-aa62-4bb8-90db-c5ee9518b194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.791903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.780385 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-kube-api-access-84888" pod "f593f01e-a1ad-4ebd-b605-af6e203a8b7e" (UID: "f593f01e-a1ad-4ebd-b605-af6e203a8b7e"). InnerVolumeSpecName "kube-api-access-84888". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.867443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.867407 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.867443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.867439 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf8xn\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-kube-api-access-sf8xn\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.867443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.867451 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84888\" (UniqueName: \"kubernetes.io/projected/f593f01e-a1ad-4ebd-b605-af6e203a8b7e-kube-api-access-84888\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.867662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.867460 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknx8\" (UniqueName: \"kubernetes.io/projected/d6f141fd-aa62-4bb8-90db-c5ee9518b194-kube-api-access-hknx8\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.867662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.867469 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4df3520-364b-4282-a066-a83c7e2ff592-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.867662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.867478 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4df3520-364b-4282-a066-a83c7e2ff592-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.867662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.867487 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.867662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.867500 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f141fd-aa62-4bb8-90db-c5ee9518b194-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.867662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.867513 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4df3520-364b-4282-a066-a83c7e2ff592-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.945165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.944959 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:22.969392 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.969354 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:22.971915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:22.971886 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:23.054495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.054416 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j"] Nov 02 15:27:23.065735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.065697 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-zt76j"] Nov 02 15:27:23.205506 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.205476 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:23.504885 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.504855 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" Nov 02 15:27:23.505060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.504891 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh" event={"ID":"f593f01e-a1ad-4ebd-b605-af6e203a8b7e","Type":"ContainerDied","Data":"d690a9227c5d74e532853f3167a2021188a30e03b263b0d86ba5ca1653491b2b"} Nov 02 15:27:23.505060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.504950 5433 scope.go:122] "RemoveContainer" containerID="054d4d6a9dbe8bd7b7e9a5cbd17da346e3a0b9b7316357d0e4d674b996aa7eaa" Nov 02 15:27:23.506674 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.506653 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" Nov 02 15:27:23.506792 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.506687 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc" event={"ID":"d4df3520-364b-4282-a066-a83c7e2ff592","Type":"ContainerDied","Data":"53c0c13071dad66865c813314c681c0b7c034ffc3670c68218a0e0a4caab88cb"} Nov 02 15:27:23.509526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.509501 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" event={"ID":"d6f141fd-aa62-4bb8-90db-c5ee9518b194","Type":"ContainerDied","Data":"55e72edde7cfc1ca42348519597874fa1a884ba98ab2c97af7cf53ae0b8ebba0"} Nov 02 15:27:23.509610 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.509574 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc" Nov 02 15:27:23.517177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.517128 5433 scope.go:122] "RemoveContainer" containerID="99958702c4af86ad6a260f9483094fdc0358338baf8429814b112f218bb879b8" Nov 02 15:27:23.527838 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.527818 5433 scope.go:122] "RemoveContainer" containerID="db34fca964b0aac202e2f6ccb8dfc5a5c2c018e00dac826fff17cd951fd66b17" Nov 02 15:27:23.581591 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.581555 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb"] Nov 02 15:27:23.583202 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:27:23.583175 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e95535_82c8_4251_80d3_da54d4f1c6d2.slice/crio-32740835bd34c08bb2b35cca7f613d11d63ad87291036ff31d440ad8de1cbfd2 WatchSource:0}: Error finding container 32740835bd34c08bb2b35cca7f613d11d63ad87291036ff31d440ad8de1cbfd2: Status 404 returned error can't find the container with id 32740835bd34c08bb2b35cca7f613d11d63ad87291036ff31d440ad8de1cbfd2 Nov 02 15:27:23.662957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.662922 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc"] Nov 02 15:27:23.667440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.667407 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-4lqzc"] Nov 02 15:27:23.770429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.770333 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh"] Nov 02 15:27:23.774435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.774405 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-5cdrh"] Nov 02 15:27:23.877324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.877289 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc"] Nov 02 15:27:23.882707 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:23.882675 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-dv9fc"] Nov 02 15:27:24.458051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:24.458021 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" path="/var/lib/kubelet/pods/26cf8c8a-484a-42d7-b251-b4950a0c052a/volumes" Nov 02 15:27:24.458531 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:24.458517 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" path="/var/lib/kubelet/pods/d4df3520-364b-4282-a066-a83c7e2ff592/volumes" Nov 02 15:27:24.458957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:24.458945 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" path="/var/lib/kubelet/pods/d6f141fd-aa62-4bb8-90db-c5ee9518b194/volumes" Nov 02 15:27:24.459722 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:24.459707 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" path="/var/lib/kubelet/pods/f593f01e-a1ad-4ebd-b605-af6e203a8b7e/volumes" Nov 02 15:27:24.522402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:24.522377 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" event={"ID":"95e95535-82c8-4251-80d3-da54d4f1c6d2","Type":"ContainerStarted","Data":"32740835bd34c08bb2b35cca7f613d11d63ad87291036ff31d440ad8de1cbfd2"} Nov 02 15:27:24.847262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:24.847187 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch"] Nov 02 15:27:24.948313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:24.948282 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x"] Nov 02 15:27:25.192201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:25.192164 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:25.335998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:25.330942 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:25.558049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:25.557887 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" event={"ID":"95e95535-82c8-4251-80d3-da54d4f1c6d2","Type":"ContainerStarted","Data":"7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2"} Nov 02 15:27:25.558524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:25.558140 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:26.063317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:26.063272 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:28.068177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:28.068145 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:29.848365 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:29.848321 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" podUID="85b21472-eadd-4945-bae1-dc45cdc075c6" containerName="payload-processing" containerID="cri-o://49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640" gracePeriod=25 Nov 02 15:27:29.948722 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:29.948673 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" podUID="a54d492f-3f09-4f5e-bed2-9f4394ec9db7" containerName="payload-pre-processing" containerID="cri-o://28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c" gracePeriod=25 Nov 02 15:27:30.143297 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.143269 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:27:30.253721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.253657 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" podStartSLOduration=7.381892848 podStartE2EDuration="8.25363733s" totalImagesPullingTime="871.744482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:25.689293455 +0000 UTC m=+1749.362496652" watchObservedRunningTime="2026-11-02 15:27:30.25363733 +0000 UTC m=+1753.926840525" Nov 02 15:27:30.257954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.257926 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:27:30.306802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.306771 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b21472-eadd-4945-bae1-dc45cdc075c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b21472-eadd-4945-bae1-dc45cdc075c6-plugins-config-volume\") pod \"85b21472-eadd-4945-bae1-dc45cdc075c6\" (UID: \"85b21472-eadd-4945-bae1-dc45cdc075c6\") " Nov 02 15:27:30.307067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.307044 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b21472-eadd-4945-bae1-dc45cdc075c6-kube-api-access-jcfb2\" (UniqueName: \"kubernetes.io/projected/85b21472-eadd-4945-bae1-dc45cdc075c6-kube-api-access-jcfb2\") pod \"85b21472-eadd-4945-bae1-dc45cdc075c6\" (UID: \"85b21472-eadd-4945-bae1-dc45cdc075c6\") " Nov 02 15:27:30.307523 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.307218 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b21472-eadd-4945-bae1-dc45cdc075c6-plugins-config-volume" pod "85b21472-eadd-4945-bae1-dc45cdc075c6" (UID: "85b21472-eadd-4945-bae1-dc45cdc075c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:30.307643 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.307618 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b21472-eadd-4945-bae1-dc45cdc075c6-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:30.309503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.309474 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b21472-eadd-4945-bae1-dc45cdc075c6-kube-api-access-jcfb2" pod "85b21472-eadd-4945-bae1-dc45cdc075c6" (UID: "85b21472-eadd-4945-bae1-dc45cdc075c6"). InnerVolumeSpecName "kube-api-access-jcfb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:30.409174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.409145 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-plugins-config-volume\") pod \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\" (UID: \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\") " Nov 02 15:27:30.409331 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.409199 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-kube-api-access-4dttf\" (UniqueName: \"kubernetes.io/projected/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-kube-api-access-4dttf\") pod \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\" (UID: \"a54d492f-3f09-4f5e-bed2-9f4394ec9db7\") " Nov 02 15:27:30.409590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.409565 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-plugins-config-volume" pod "a54d492f-3f09-4f5e-bed2-9f4394ec9db7" (UID: "a54d492f-3f09-4f5e-bed2-9f4394ec9db7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:30.409720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.409702 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:30.409768 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.409723 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcfb2\" (UniqueName: \"kubernetes.io/projected/85b21472-eadd-4945-bae1-dc45cdc075c6-kube-api-access-jcfb2\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:30.411672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.411646 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-kube-api-access-4dttf" pod "a54d492f-3f09-4f5e-bed2-9f4394ec9db7" (UID: "a54d492f-3f09-4f5e-bed2-9f4394ec9db7"). InnerVolumeSpecName "kube-api-access-4dttf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:30.511181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.511148 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dttf\" (UniqueName: \"kubernetes.io/projected/a54d492f-3f09-4f5e-bed2-9f4394ec9db7-kube-api-access-4dttf\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:30.604760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.604724 5433 generic.go:356] "Generic (PLEG): container finished" podID="a54d492f-3f09-4f5e-bed2-9f4394ec9db7" containerID="28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c" exitCode=0 Nov 02 15:27:30.604909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.604766 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" Nov 02 15:27:30.604909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.604805 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" event={"ID":"a54d492f-3f09-4f5e-bed2-9f4394ec9db7","Type":"ContainerDied","Data":"28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c"} Nov 02 15:27:30.604909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.604858 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x" event={"ID":"a54d492f-3f09-4f5e-bed2-9f4394ec9db7","Type":"ContainerDied","Data":"00d62a0e0b528a4fb7692c1bc7911182b3d3b6af0894b2e6bda2098f33d05cb5"} Nov 02 15:27:30.604909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.604874 5433 scope.go:122] "RemoveContainer" containerID="28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c" Nov 02 15:27:30.606450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.606425 5433 generic.go:356] "Generic (PLEG): container finished" podID="85b21472-eadd-4945-bae1-dc45cdc075c6" containerID="49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640" exitCode=0 Nov 02 15:27:30.606568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.606481 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" Nov 02 15:27:30.606568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.606514 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" event={"ID":"85b21472-eadd-4945-bae1-dc45cdc075c6","Type":"ContainerDied","Data":"49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640"} Nov 02 15:27:30.606678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.606577 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch" event={"ID":"85b21472-eadd-4945-bae1-dc45cdc075c6","Type":"ContainerDied","Data":"220cd58f1b7f55048f11bcc00ad65e353d8011c4d6303284353fdf796c488ffb"} Nov 02 15:27:30.615767 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.615749 5433 scope.go:122] "RemoveContainer" containerID="28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c" Nov 02 15:27:30.616073 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:30.616051 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c\": container with ID starting with 28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c not found: ID does not exist" containerID="28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c" Nov 02 15:27:30.616120 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.616082 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c"} err="failed to get container status \"28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c\": rpc error: code = NotFound desc = could not find container \"28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c\": container with ID starting with 28080e1457515cd3a7a1cdb54475bd245c0f23ad5986e2ab66de5c5523e3eb2c not found: ID does not exist" Nov 02 15:27:30.616120 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.616098 5433 scope.go:122] "RemoveContainer" containerID="49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640" Nov 02 15:27:30.624961 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.624943 5433 scope.go:122] "RemoveContainer" containerID="49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640" Nov 02 15:27:30.625235 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:30.625215 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640\": container with ID starting with 49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640 not found: ID does not exist" containerID="49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640" Nov 02 15:27:30.625287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.625243 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640"} err="failed to get container status \"49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640\": rpc error: code = NotFound desc = could not find container \"49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640\": container with ID starting with 49aaaade69d0ca98a24bc34d77e9515a1748d9ecd4d9d1022745d7fc3596d640 not found: ID does not exist" Nov 02 15:27:30.670727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.670669 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:30.758782 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.758755 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x"] Nov 02 15:27:30.804806 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.804782 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0b05fd-6d599d4b87-rs46x"] Nov 02 15:27:30.864635 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.864607 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch"] Nov 02 15:27:30.869099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:30.869078 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9d0b05fd-7ff5898bc-rn9ch"] Nov 02 15:27:31.581435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:31.581407 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:27:32.458562 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:32.458464 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b21472-eadd-4945-bae1-dc45cdc075c6" path="/var/lib/kubelet/pods/85b21472-eadd-4945-bae1-dc45cdc075c6/volumes" Nov 02 15:27:32.459188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:32.459165 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54d492f-3f09-4f5e-bed2-9f4394ec9db7" path="/var/lib/kubelet/pods/a54d492f-3f09-4f5e-bed2-9f4394ec9db7/volumes" Nov 02 15:27:33.208257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:33.208209 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:35.828309 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:35.828276 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:35.976335 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:35.976296 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:36.541247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:36.541195 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:36.718896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:36.718856 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:38.093721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.093661 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:38.188645 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.188614 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q"] Nov 02 15:27:38.189331 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189315 5433 cpu_manager.go:414] "containerMap: removing container" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" Nov 02 15:27:38.189386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189333 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" Nov 02 15:27:38.189386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189346 5433 cpu_manager.go:414] "containerMap: removing container" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" Nov 02 15:27:38.189386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189352 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" Nov 02 15:27:38.189386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189359 5433 cpu_manager.go:414] "containerMap: removing container" podUID="85b21472-eadd-4945-bae1-dc45cdc075c6" containerName="payload-processing" Nov 02 15:27:38.189386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189367 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b21472-eadd-4945-bae1-dc45cdc075c6" containerName="payload-processing" Nov 02 15:27:38.189386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189375 5433 cpu_manager.go:414] "containerMap: removing container" podUID="a54d492f-3f09-4f5e-bed2-9f4394ec9db7" containerName="payload-pre-processing" Nov 02 15:27:38.189386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189380 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54d492f-3f09-4f5e-bed2-9f4394ec9db7" containerName="payload-pre-processing" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189394 5433 cpu_manager.go:414] "containerMap: removing container" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189399 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189408 5433 cpu_manager.go:414] "containerMap: removing container" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189412 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189420 5433 cpu_manager.go:414] "containerMap: removing container" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189424 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189540 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f141fd-aa62-4bb8-90db-c5ee9518b194" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189549 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189554 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b21472-eadd-4945-bae1-dc45cdc075c6" containerName="payload-processing" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189560 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54d492f-3f09-4f5e-bed2-9f4394ec9db7" containerName="payload-pre-processing" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189566 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="f593f01e-a1ad-4ebd-b605-af6e203a8b7e" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189573 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cf8c8a-484a-42d7-b251-b4950a0c052a" containerName="istio-proxy" Nov 02 15:27:38.189592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.189579 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4df3520-364b-4282-a066-a83c7e2ff592" containerName="istio-proxy" Nov 02 15:27:38.192856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.192837 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.251436 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.251393 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.251627 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.251552 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phpmd\" (UniqueName: \"kubernetes.io/projected/b2aa573f-e7f4-4b63-9615-98072075a183-kube-api-access-phpmd\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.251797 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.251767 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aa573f-e7f4-4b63-9615-98072075a183-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.252191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.252161 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.289928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.289891 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-7ad552-serving-cert\"" Nov 02 15:27:38.293868 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.293836 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q"] Nov 02 15:27:38.354442 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.354349 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.354619 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.354477 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phpmd\" (UniqueName: \"kubernetes.io/projected/b2aa573f-e7f4-4b63-9615-98072075a183-kube-api-access-phpmd\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.355076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.354621 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aa573f-e7f4-4b63-9615-98072075a183-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.355076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.354785 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.355076 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:38.354940 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-serving-cert: secret "maas-api-e2e-shared-a-w2-7ad552-serving-cert" not found Nov 02 15:27:38.355076 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:38.355024 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls podName:b2aa573f-e7f4-4b63-9615-98072075a183 nodeName:}" failed. No retries permitted until 2026-11-02 15:27:38.855000739 +0000 UTC m=+1762.528203938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls") pod "maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" (UID: "b2aa573f-e7f4-4b63-9615-98072075a183") : secret "maas-api-e2e-shared-a-w2-7ad552-serving-cert" not found Nov 02 15:27:38.355812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.355782 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aa573f-e7f4-4b63-9615-98072075a183-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.358465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.358437 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.389304 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.389271 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:38.417479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.417452 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phpmd\" (UniqueName: \"kubernetes.io/projected/b2aa573f-e7f4-4b63-9615-98072075a183-kube-api-access-phpmd\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.591045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.585008 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:38.758717 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.758583 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:38.878905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.878865 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:38.881317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:38.881287 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:39.106624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:39.106535 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:39.486897 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:27:39.486866 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2aa573f_e7f4_4b63_9615_98072075a183.slice/crio-7120006c50dcdde5397e728107ff612663196564aff05137cc18543bda18a32b WatchSource:0}: Error finding container 7120006c50dcdde5397e728107ff612663196564aff05137cc18543bda18a32b: Status 404 returned error can't find the container with id 7120006c50dcdde5397e728107ff612663196564aff05137cc18543bda18a32b Nov 02 15:27:39.488103 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:39.488080 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q"] Nov 02 15:27:39.688923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:39.688890 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" event={"ID":"b2aa573f-e7f4-4b63-9615-98072075a183","Type":"ContainerStarted","Data":"7120006c50dcdde5397e728107ff612663196564aff05137cc18543bda18a32b"} Nov 02 15:27:40.698607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:40.698572 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" event={"ID":"b2aa573f-e7f4-4b63-9615-98072075a183","Type":"ContainerStarted","Data":"330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28"} Nov 02 15:27:40.699043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:40.698668 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:40.811830 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:40.811774 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" podStartSLOduration=1.936195855 podStartE2EDuration="2.811758634s" totalImagesPullingTime="875.562779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:40.808787718 +0000 UTC m=+1764.481990913" watchObservedRunningTime="2026-11-02 15:27:40.811758634 +0000 UTC m=+1764.484961832" Nov 02 15:27:41.077794 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:41.077706 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:27:43.769680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:43.769639 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:44.547403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.547366 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm"] Nov 02 15:27:44.547613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.547578 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" containerID="cri-o://2fa2d3898c42c9114ebd820f1e256c3032b56bdc3cb28776e98f96e05fea0d32" gracePeriod=30 Nov 02 15:27:44.593559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.593522 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn"] Nov 02 15:27:44.593781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.593758 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" containerID="cri-o://99219287e538959f94e4b7f65fa2ca3da7c15596891f147a6db886fb93d5d336" gracePeriod=30 Nov 02 15:27:44.596684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.596654 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m"] Nov 02 15:27:44.596880 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.596848 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" containerID="cri-o://2ab6bee5b2a31ec4f0f84997b17c69e36301c1aaaa88f09d31eab9daed2a5c61" gracePeriod=30 Nov 02 15:27:44.640332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.640294 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv"] Nov 02 15:27:44.640555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.640516 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" containerID="cri-o://0782cea2eb169be3541c07639f7c10cdd0c2502f349fde5a5e405f053a079fdd" gracePeriod=30 Nov 02 15:27:44.645413 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.645375 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj"] Nov 02 15:27:44.645646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:44.645616 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" containerID="cri-o://1ed2b2efdc468e66a9628e034a3be9e9a578c7db31bf8a431ce7d7bf488b04cb" gracePeriod=30 Nov 02 15:27:46.715288 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:46.715257 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:27:47.232361 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.232301 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp"] Nov 02 15:27:47.232582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.232557 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" containerID="cri-o://bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4" gracePeriod=30 Nov 02 15:27:47.278519 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.278476 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf"] Nov 02 15:27:47.278749 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.278723 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" containerID="cri-o://fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d" gracePeriod=30 Nov 02 15:27:47.282405 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.282366 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r"] Nov 02 15:27:47.282695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.282642 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" containerID="cri-o://c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46" gracePeriod=30 Nov 02 15:27:47.286027 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.286003 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv"] Nov 02 15:27:47.286220 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.286195 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" containerID="cri-o://3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3" gracePeriod=30 Nov 02 15:27:47.515595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.515504 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:47.784253 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.784094 5433 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:27:47.784253 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.784155 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:27:47.805618 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:47.805571 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:49.072077 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.072046 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:49.771391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.771354 5433 generic.go:356] "Generic (PLEG): container finished" podID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerID="2ab6bee5b2a31ec4f0f84997b17c69e36301c1aaaa88f09d31eab9daed2a5c61" exitCode=0 Nov 02 15:27:49.771505 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.771453 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" event={"ID":"26f9d780-9ea7-4084-8636-b99d44a79c3f","Type":"ContainerDied","Data":"2ab6bee5b2a31ec4f0f84997b17c69e36301c1aaaa88f09d31eab9daed2a5c61"} Nov 02 15:27:49.791132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.773497 5433 generic.go:356] "Generic (PLEG): container finished" podID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerID="1ed2b2efdc468e66a9628e034a3be9e9a578c7db31bf8a431ce7d7bf488b04cb" exitCode=0 Nov 02 15:27:49.791132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.773574 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" event={"ID":"92a2957d-3b26-49a9-862c-6ff5d83cb626","Type":"ContainerDied","Data":"1ed2b2efdc468e66a9628e034a3be9e9a578c7db31bf8a431ce7d7bf488b04cb"} Nov 02 15:27:49.791132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.775234 5433 generic.go:356] "Generic (PLEG): container finished" podID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerID="2fa2d3898c42c9114ebd820f1e256c3032b56bdc3cb28776e98f96e05fea0d32" exitCode=0 Nov 02 15:27:49.791132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.775390 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" event={"ID":"86cca276-7d0e-4481-a1d6-d6931fdac725","Type":"ContainerDied","Data":"2fa2d3898c42c9114ebd820f1e256c3032b56bdc3cb28776e98f96e05fea0d32"} Nov 02 15:27:49.791132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.776797 5433 generic.go:356] "Generic (PLEG): container finished" podID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerID="99219287e538959f94e4b7f65fa2ca3da7c15596891f147a6db886fb93d5d336" exitCode=0 Nov 02 15:27:49.791132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.776867 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" event={"ID":"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1","Type":"ContainerDied","Data":"99219287e538959f94e4b7f65fa2ca3da7c15596891f147a6db886fb93d5d336"} Nov 02 15:27:49.791132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.778671 5433 generic.go:356] "Generic (PLEG): container finished" podID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerID="0782cea2eb169be3541c07639f7c10cdd0c2502f349fde5a5e405f053a079fdd" exitCode=0 Nov 02 15:27:49.791132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.778718 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" event={"ID":"e411e1cf-c1c8-4d31-84fd-91a4d0167b86","Type":"ContainerDied","Data":"0782cea2eb169be3541c07639f7c10cdd0c2502f349fde5a5e405f053a079fdd"} Nov 02 15:27:49.903644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.903608 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:27:49.948248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948205 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-socket\") pod \"86cca276-7d0e-4481-a1d6-d6931fdac725\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " Nov 02 15:27:49.948453 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948317 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86cca276-7d0e-4481-a1d6-d6931fdac725-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cca276-7d0e-4481-a1d6-d6931fdac725-istiod-ca-cert\") pod \"86cca276-7d0e-4481-a1d6-d6931fdac725\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " Nov 02 15:27:49.948453 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948417 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-data\") pod \"86cca276-7d0e-4481-a1d6-d6931fdac725\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " Nov 02 15:27:49.948569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948474 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-credential-socket\") pod \"86cca276-7d0e-4481-a1d6-d6931fdac725\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " Nov 02 15:27:49.948569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948505 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-socket" pod "86cca276-7d0e-4481-a1d6-d6931fdac725" (UID: "86cca276-7d0e-4481-a1d6-d6931fdac725"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:49.948569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948515 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-envoy\") pod \"86cca276-7d0e-4481-a1d6-d6931fdac725\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " Nov 02 15:27:49.948725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948581 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-token\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-token\") pod \"86cca276-7d0e-4481-a1d6-d6931fdac725\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " Nov 02 15:27:49.948725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948617 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-podinfo\") pod \"86cca276-7d0e-4481-a1d6-d6931fdac725\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " Nov 02 15:27:49.948826 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948727 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-data" pod "86cca276-7d0e-4481-a1d6-d6931fdac725" (UID: "86cca276-7d0e-4481-a1d6-d6931fdac725"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:49.948826 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948732 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-certs\") pod \"86cca276-7d0e-4481-a1d6-d6931fdac725\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " Nov 02 15:27:49.948826 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948780 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cca276-7d0e-4481-a1d6-d6931fdac725-istiod-ca-cert" pod "86cca276-7d0e-4481-a1d6-d6931fdac725" (UID: "86cca276-7d0e-4481-a1d6-d6931fdac725"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:49.949018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.948866 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-kube-api-access-dkcrz\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-kube-api-access-dkcrz\") pod \"86cca276-7d0e-4481-a1d6-d6931fdac725\" (UID: \"86cca276-7d0e-4481-a1d6-d6931fdac725\") " Nov 02 15:27:49.949438 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.949180 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-certs" pod "86cca276-7d0e-4481-a1d6-d6931fdac725" (UID: "86cca276-7d0e-4481-a1d6-d6931fdac725"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:49.949438 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.949249 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-credential-socket" pod "86cca276-7d0e-4481-a1d6-d6931fdac725" (UID: "86cca276-7d0e-4481-a1d6-d6931fdac725"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:49.949889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.949859 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:49.950013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.949895 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cca276-7d0e-4481-a1d6-d6931fdac725-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:49.950013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.949911 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:49.950013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.949924 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:49.950013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.949938 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:49.951993 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.951943 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-envoy" pod "86cca276-7d0e-4481-a1d6-d6931fdac725" (UID: "86cca276-7d0e-4481-a1d6-d6931fdac725"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:49.952094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.952011 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-kube-api-access-dkcrz" pod "86cca276-7d0e-4481-a1d6-d6931fdac725" (UID: "86cca276-7d0e-4481-a1d6-d6931fdac725"). InnerVolumeSpecName "kube-api-access-dkcrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:49.953073 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.953047 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-podinfo" pod "86cca276-7d0e-4481-a1d6-d6931fdac725" (UID: "86cca276-7d0e-4481-a1d6-d6931fdac725"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:49.953269 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:49.953246 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-token" pod "86cca276-7d0e-4481-a1d6-d6931fdac725" (UID: "86cca276-7d0e-4481-a1d6-d6931fdac725"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.061836 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.061730 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkcrz\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-kube-api-access-dkcrz\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.061836 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.061770 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.061836 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.061788 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.061836 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.061802 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cca276-7d0e-4481-a1d6-d6931fdac725-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.092880 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.092834 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:27:50.097744 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.097675 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:27:50.107841 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.107814 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:27:50.110164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.110137 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:27:50.162756 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.162712 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-token\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-token\") pod \"26f9d780-9ea7-4084-8636-b99d44a79c3f\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " Nov 02 15:27:50.162985 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.162821 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-kube-api-access-9m6f4\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-kube-api-access-9m6f4\") pod \"26f9d780-9ea7-4084-8636-b99d44a79c3f\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " Nov 02 15:27:50.163257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.162858 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-podinfo\") pod \"92a2957d-3b26-49a9-862c-6ff5d83cb626\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " Nov 02 15:27:50.163356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.163334 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-envoy\") pod \"26f9d780-9ea7-4084-8636-b99d44a79c3f\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " Nov 02 15:27:50.163729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.163705 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-token\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-token\") pod \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " Nov 02 15:27:50.164828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.164804 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-credential-socket\") pod \"26f9d780-9ea7-4084-8636-b99d44a79c3f\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " Nov 02 15:27:50.165187 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.165129 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-credential-socket" pod "26f9d780-9ea7-4084-8636-b99d44a79c3f" (UID: "26f9d780-9ea7-4084-8636-b99d44a79c3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.165371 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.165352 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-data\") pod \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " Nov 02 15:27:50.165828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.165729 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-data" pod "e411e1cf-c1c8-4d31-84fd-91a4d0167b86" (UID: "e411e1cf-c1c8-4d31-84fd-91a4d0167b86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.166258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166233 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-envoy\") pod \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " Nov 02 15:27:50.166367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166275 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-certs\") pod \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " Nov 02 15:27:50.166367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166311 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-data\") pod \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " Nov 02 15:27:50.166485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166382 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istiod-ca-cert\") pod \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " Nov 02 15:27:50.166485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166440 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-certs\") pod \"92a2957d-3b26-49a9-862c-6ff5d83cb626\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " Nov 02 15:27:50.166584 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166492 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-token\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-token\") pod \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " Nov 02 15:27:50.166584 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166543 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-socket\") pod \"92a2957d-3b26-49a9-862c-6ff5d83cb626\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " Nov 02 15:27:50.166681 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166571 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-certs\") pod \"26f9d780-9ea7-4084-8636-b99d44a79c3f\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " Nov 02 15:27:50.166681 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166640 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-podinfo\") pod \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " Nov 02 15:27:50.167582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166719 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-kube-api-access-mmxbg\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-kube-api-access-mmxbg\") pod \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " Nov 02 15:27:50.167582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.166958 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-certs" pod "26f9d780-9ea7-4084-8636-b99d44a79c3f" (UID: "26f9d780-9ea7-4084-8636-b99d44a79c3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.167582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.167299 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-kube-api-access-nk7xz\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-kube-api-access-nk7xz\") pod \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " Nov 02 15:27:50.167582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.167360 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istiod-ca-cert\") pod \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " Nov 02 15:27:50.168012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.167798 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-data" pod "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" (UID: "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.168012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.167832 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-certs" pod "92a2957d-3b26-49a9-862c-6ff5d83cb626" (UID: "92a2957d-3b26-49a9-862c-6ff5d83cb626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.168432 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.168300 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-socket" pod "92a2957d-3b26-49a9-862c-6ff5d83cb626" (UID: "92a2957d-3b26-49a9-862c-6ff5d83cb626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.168432 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.168399 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istiod-ca-cert" pod "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" (UID: "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:50.168577 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.168483 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istiod-ca-cert" pod "e411e1cf-c1c8-4d31-84fd-91a4d0167b86" (UID: "e411e1cf-c1c8-4d31-84fd-91a4d0167b86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:50.168705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.168668 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-certs" pod "e411e1cf-c1c8-4d31-84fd-91a4d0167b86" (UID: "e411e1cf-c1c8-4d31-84fd-91a4d0167b86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.168840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.168823 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-data\") pod \"26f9d780-9ea7-4084-8636-b99d44a79c3f\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " Nov 02 15:27:50.170317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.169171 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-data" pod "26f9d780-9ea7-4084-8636-b99d44a79c3f" (UID: "26f9d780-9ea7-4084-8636-b99d44a79c3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.170719 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.169237 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-envoy\") pod \"92a2957d-3b26-49a9-862c-6ff5d83cb626\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " Nov 02 15:27:50.170719 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.170638 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-socket\") pod \"26f9d780-9ea7-4084-8636-b99d44a79c3f\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " Nov 02 15:27:50.170942 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.170872 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-socket" pod "26f9d780-9ea7-4084-8636-b99d44a79c3f" (UID: "26f9d780-9ea7-4084-8636-b99d44a79c3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.171022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.170946 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f9d780-9ea7-4084-8636-b99d44a79c3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f9d780-9ea7-4084-8636-b99d44a79c3f-istiod-ca-cert\") pod \"26f9d780-9ea7-4084-8636-b99d44a79c3f\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " Nov 02 15:27:50.171274 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.171250 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f9d780-9ea7-4084-8636-b99d44a79c3f-istiod-ca-cert" pod "26f9d780-9ea7-4084-8636-b99d44a79c3f" (UID: "26f9d780-9ea7-4084-8636-b99d44a79c3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:50.172584 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.172468 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-token" pod "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" (UID: "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.172760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.172680 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-socket\") pod \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " Nov 02 15:27:50.172760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.172722 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-socket" pod "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" (UID: "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.172760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.172739 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-podinfo\") pod \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " Nov 02 15:27:50.173131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.173107 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-token" pod "26f9d780-9ea7-4084-8636-b99d44a79c3f" (UID: "26f9d780-9ea7-4084-8636-b99d44a79c3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.173294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.173256 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-kube-api-access-nk7xz" pod "e411e1cf-c1c8-4d31-84fd-91a4d0167b86" (UID: "e411e1cf-c1c8-4d31-84fd-91a4d0167b86"). InnerVolumeSpecName "kube-api-access-nk7xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.178682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178534 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-certs\") pod \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " Nov 02 15:27:50.178682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178580 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-envoy" pod "e411e1cf-c1c8-4d31-84fd-91a4d0167b86" (UID: "e411e1cf-c1c8-4d31-84fd-91a4d0167b86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.178682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178588 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-token" pod "e411e1cf-c1c8-4d31-84fd-91a4d0167b86" (UID: "e411e1cf-c1c8-4d31-84fd-91a4d0167b86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.178682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178588 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-envoy\") pod \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " Nov 02 15:27:50.178682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178630 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-kube-api-access-mmxbg" pod "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" (UID: "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1"). InnerVolumeSpecName "kube-api-access-mmxbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.178682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178670 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-token\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-token\") pod \"92a2957d-3b26-49a9-862c-6ff5d83cb626\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178703 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-data\") pod \"92a2957d-3b26-49a9-862c-6ff5d83cb626\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178731 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-podinfo\") pod \"26f9d780-9ea7-4084-8636-b99d44a79c3f\" (UID: \"26f9d780-9ea7-4084-8636-b99d44a79c3f\") " Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178825 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-certs" pod "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" (UID: "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.178857 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-podinfo" pod "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" (UID: "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179143 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-credential-socket\") pod \"92a2957d-3b26-49a9-862c-6ff5d83cb626\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179204 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-kube-api-access-vq9hm\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-kube-api-access-vq9hm\") pod \"92a2957d-3b26-49a9-862c-6ff5d83cb626\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179246 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-credential-socket\") pod \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179284 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a2957d-3b26-49a9-862c-6ff5d83cb626-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2957d-3b26-49a9-862c-6ff5d83cb626-istiod-ca-cert\") pod \"92a2957d-3b26-49a9-862c-6ff5d83cb626\" (UID: \"92a2957d-3b26-49a9-862c-6ff5d83cb626\") " Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179316 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-socket\") pod \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\" (UID: \"e411e1cf-c1c8-4d31-84fd-91a4d0167b86\") " Nov 02 15:27:50.179425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179364 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-credential-socket\") pod \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\" (UID: \"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1\") " Nov 02 15:27:50.180379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179027 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-data" pod "92a2957d-3b26-49a9-862c-6ff5d83cb626" (UID: "92a2957d-3b26-49a9-862c-6ff5d83cb626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.180379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179354 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-credential-socket" pod "92a2957d-3b26-49a9-862c-6ff5d83cb626" (UID: "92a2957d-3b26-49a9-862c-6ff5d83cb626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.180379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179409 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-envoy" pod "26f9d780-9ea7-4084-8636-b99d44a79c3f" (UID: "26f9d780-9ea7-4084-8636-b99d44a79c3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.180379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179451 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-envoy" pod "92a2957d-3b26-49a9-862c-6ff5d83cb626" (UID: "92a2957d-3b26-49a9-862c-6ff5d83cb626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.180379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179701 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-kube-api-access-9m6f4" pod "26f9d780-9ea7-4084-8636-b99d44a79c3f" (UID: "26f9d780-9ea7-4084-8636-b99d44a79c3f"). InnerVolumeSpecName "kube-api-access-9m6f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.180379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.179749 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-podinfo" pod "92a2957d-3b26-49a9-862c-6ff5d83cb626" (UID: "92a2957d-3b26-49a9-862c-6ff5d83cb626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:50.180379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.180086 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a2957d-3b26-49a9-862c-6ff5d83cb626-istiod-ca-cert" pod "92a2957d-3b26-49a9-862c-6ff5d83cb626" (UID: "92a2957d-3b26-49a9-862c-6ff5d83cb626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:50.180812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.180363 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-credential-socket" pod "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" (UID: "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.180812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.180600 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-socket" pod "e411e1cf-c1c8-4d31-84fd-91a4d0167b86" (UID: "e411e1cf-c1c8-4d31-84fd-91a4d0167b86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181085 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181109 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181129 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181145 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181156 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181164 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxbg\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-kube-api-access-mmxbg\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181173 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk7xz\" (UniqueName: \"kubernetes.io/projected/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-kube-api-access-nk7xz\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181181 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181192 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181191 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-token" pod "92a2957d-3b26-49a9-862c-6ff5d83cb626" (UID: "92a2957d-3b26-49a9-862c-6ff5d83cb626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181204 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181218 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181230 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f9d780-9ea7-4084-8636-b99d44a79c3f-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181244 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181256 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181268 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181282 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a2957d-3b26-49a9-862c-6ff5d83cb626-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181294 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a2957d-3b26-49a9-862c-6ff5d83cb626-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181306 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181318 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181331 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181345 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m6f4\" (UniqueName: \"kubernetes.io/projected/26f9d780-9ea7-4084-8636-b99d44a79c3f-kube-api-access-9m6f4\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181357 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181369 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181381 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181395 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f9d780-9ea7-4084-8636-b99d44a79c3f-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181407 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181420 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181433 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181446 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.181478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181458 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.182956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.181642 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-podinfo" pod "e411e1cf-c1c8-4d31-84fd-91a4d0167b86" (UID: "e411e1cf-c1c8-4d31-84fd-91a4d0167b86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:50.182956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.182693 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-kube-api-access-vq9hm" pod "92a2957d-3b26-49a9-862c-6ff5d83cb626" (UID: "92a2957d-3b26-49a9-862c-6ff5d83cb626"). InnerVolumeSpecName "kube-api-access-vq9hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.182956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.182927 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-envoy" pod "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" (UID: "10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.183138 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.183075 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-credential-socket" pod "e411e1cf-c1c8-4d31-84fd-91a4d0167b86" (UID: "e411e1cf-c1c8-4d31-84fd-91a4d0167b86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.183190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.183168 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-podinfo" pod "26f9d780-9ea7-4084-8636-b99d44a79c3f" (UID: "26f9d780-9ea7-4084-8636-b99d44a79c3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:50.283054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.283016 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.283054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.283047 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.283054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.283060 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.283297 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.283070 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f9d780-9ea7-4084-8636-b99d44a79c3f-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.283297 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.283080 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq9hm\" (UniqueName: \"kubernetes.io/projected/92a2957d-3b26-49a9-862c-6ff5d83cb626-kube-api-access-vq9hm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.283297 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.283090 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e411e1cf-c1c8-4d31-84fd-91a4d0167b86-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.788588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.788495 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" Nov 02 15:27:50.788588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.788533 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv" event={"ID":"e411e1cf-c1c8-4d31-84fd-91a4d0167b86","Type":"ContainerDied","Data":"288bb543a2f29ae92973e6631eeb8fa5a1c28f7fe307435d948062c755fbe55e"} Nov 02 15:27:50.788807 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.788604 5433 scope.go:122] "RemoveContainer" containerID="0782cea2eb169be3541c07639f7c10cdd0c2502f349fde5a5e405f053a079fdd" Nov 02 15:27:50.790417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.790390 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" event={"ID":"26f9d780-9ea7-4084-8636-b99d44a79c3f","Type":"ContainerDied","Data":"2e1cabf63918efd3633b163ff44ea057a67d2cb2400875899b01d1ac3217e226"} Nov 02 15:27:50.790565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.790396 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m" Nov 02 15:27:50.792616 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.792575 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" event={"ID":"92a2957d-3b26-49a9-862c-6ff5d83cb626","Type":"ContainerDied","Data":"98f9b889bf711da9557e6c5e6949a28977f5c1fe8f1e19ab88c27bc000a4395e"} Nov 02 15:27:50.792761 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.792583 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj" Nov 02 15:27:50.795039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.794962 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" event={"ID":"86cca276-7d0e-4481-a1d6-d6931fdac725","Type":"ContainerDied","Data":"8e59928cfefd003ba160c65dddfe8c985eb1fd63bd6ab8662aef41cdc871659b"} Nov 02 15:27:50.795189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.795173 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm" Nov 02 15:27:50.806161 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.806134 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" event={"ID":"10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1","Type":"ContainerDied","Data":"e56edf1a86fdee79940fbd5c2b3421d45cc85d77e65c176e7db273c8d904bdf9"} Nov 02 15:27:50.806333 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.806190 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn" Nov 02 15:27:50.811237 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.811218 5433 scope.go:122] "RemoveContainer" containerID="2ab6bee5b2a31ec4f0f84997b17c69e36301c1aaaa88f09d31eab9daed2a5c61" Nov 02 15:27:50.821125 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.821100 5433 scope.go:122] "RemoveContainer" containerID="1ed2b2efdc468e66a9628e034a3be9e9a578c7db31bf8a431ce7d7bf488b04cb" Nov 02 15:27:50.831015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.830995 5433 scope.go:122] "RemoveContainer" containerID="2fa2d3898c42c9114ebd820f1e256c3032b56bdc3cb28776e98f96e05fea0d32" Nov 02 15:27:50.840933 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.840916 5433 scope.go:122] "RemoveContainer" containerID="99219287e538959f94e4b7f65fa2ca3da7c15596891f147a6db886fb93d5d336" Nov 02 15:27:50.843928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.843903 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:27:50.844049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.843953 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:27:50.946563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.946530 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj"] Nov 02 15:27:50.953244 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:50.953213 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-xssnj"] Nov 02 15:27:51.060051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.059926 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv"] Nov 02 15:27:51.065849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.065814 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-gbndv"] Nov 02 15:27:51.173898 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.173864 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m"] Nov 02 15:27:51.182642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.182609 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-6wv5m"] Nov 02 15:27:51.285028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.284967 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm"] Nov 02 15:27:51.290197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.290162 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-gv6jm"] Nov 02 15:27:51.389506 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.389468 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn"] Nov 02 15:27:51.392050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.391992 5433 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:27:51.392219 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.392078 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:27:51.396751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:51.396724 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-8pcvn"] Nov 02 15:27:52.461919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.461887 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" path="/var/lib/kubelet/pods/10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1/volumes" Nov 02 15:27:52.464736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.462599 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" path="/var/lib/kubelet/pods/26f9d780-9ea7-4084-8636-b99d44a79c3f/volumes" Nov 02 15:27:52.464736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.463094 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" path="/var/lib/kubelet/pods/86cca276-7d0e-4481-a1d6-d6931fdac725/volumes" Nov 02 15:27:52.464736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.464111 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" path="/var/lib/kubelet/pods/92a2957d-3b26-49a9-862c-6ff5d83cb626/volumes" Nov 02 15:27:52.472138 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.470818 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" path="/var/lib/kubelet/pods/e411e1cf-c1c8-4d31-84fd-91a4d0167b86/volumes" Nov 02 15:27:52.605726 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.605693 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:27:52.640860 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.640824 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-podinfo\") pod \"0b2ecad5-375c-461a-8f81-d2243d61c915\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " Nov 02 15:27:52.641084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.640930 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-certs\") pod \"0b2ecad5-375c-461a-8f81-d2243d61c915\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " Nov 02 15:27:52.641084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.640968 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b2ecad5-375c-461a-8f81-d2243d61c915-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2ecad5-375c-461a-8f81-d2243d61c915-istiod-ca-cert\") pod \"0b2ecad5-375c-461a-8f81-d2243d61c915\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " Nov 02 15:27:52.641084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.641014 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-credential-socket\") pod \"0b2ecad5-375c-461a-8f81-d2243d61c915\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " Nov 02 15:27:52.641084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.641047 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-token\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-token\") pod \"0b2ecad5-375c-461a-8f81-d2243d61c915\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " Nov 02 15:27:52.641365 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.641321 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-envoy\") pod \"0b2ecad5-375c-461a-8f81-d2243d61c915\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " Nov 02 15:27:52.641467 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.641446 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-socket\") pod \"0b2ecad5-375c-461a-8f81-d2243d61c915\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " Nov 02 15:27:52.641542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.641489 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-data\") pod \"0b2ecad5-375c-461a-8f81-d2243d61c915\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " Nov 02 15:27:52.641601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.641536 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2ecad5-375c-461a-8f81-d2243d61c915-istiod-ca-cert" pod "0b2ecad5-375c-461a-8f81-d2243d61c915" (UID: "0b2ecad5-375c-461a-8f81-d2243d61c915"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:52.641601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.641542 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-kube-api-access-d2hcc\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-kube-api-access-d2hcc\") pod \"0b2ecad5-375c-461a-8f81-d2243d61c915\" (UID: \"0b2ecad5-375c-461a-8f81-d2243d61c915\") " Nov 02 15:27:52.642512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.642332 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-certs" pod "0b2ecad5-375c-461a-8f81-d2243d61c915" (UID: "0b2ecad5-375c-461a-8f81-d2243d61c915"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.642600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.642550 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-data" pod "0b2ecad5-375c-461a-8f81-d2243d61c915" (UID: "0b2ecad5-375c-461a-8f81-d2243d61c915"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.642729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.642687 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-credential-socket" pod "0b2ecad5-375c-461a-8f81-d2243d61c915" (UID: "0b2ecad5-375c-461a-8f81-d2243d61c915"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.642729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.642702 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-socket" pod "0b2ecad5-375c-461a-8f81-d2243d61c915" (UID: "0b2ecad5-375c-461a-8f81-d2243d61c915"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.642834 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.642766 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.642834 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.642787 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2ecad5-375c-461a-8f81-d2243d61c915-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.642834 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.642801 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.643391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.643363 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-podinfo" pod "0b2ecad5-375c-461a-8f81-d2243d61c915" (UID: "0b2ecad5-375c-461a-8f81-d2243d61c915"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:52.644875 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.644847 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-token" pod "0b2ecad5-375c-461a-8f81-d2243d61c915" (UID: "0b2ecad5-375c-461a-8f81-d2243d61c915"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.645249 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.645226 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-kube-api-access-d2hcc" pod "0b2ecad5-375c-461a-8f81-d2243d61c915" (UID: "0b2ecad5-375c-461a-8f81-d2243d61c915"). InnerVolumeSpecName "kube-api-access-d2hcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.645371 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.645292 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-envoy" pod "0b2ecad5-375c-461a-8f81-d2243d61c915" (UID: "0b2ecad5-375c-461a-8f81-d2243d61c915"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.651785 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.651740 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:27:52.654556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.654529 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:27:52.658949 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.658928 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:27:52.743486 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743455 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-socket\") pod \"25c01092-3876-411c-80f1-f1edb9e90f2b\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " Nov 02 15:27:52.743678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743505 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c01092-3876-411c-80f1-f1edb9e90f2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c01092-3876-411c-80f1-f1edb9e90f2b-istiod-ca-cert\") pod \"25c01092-3876-411c-80f1-f1edb9e90f2b\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " Nov 02 15:27:52.743678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743556 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-certs\") pod \"25c01092-3876-411c-80f1-f1edb9e90f2b\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " Nov 02 15:27:52.743678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743588 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-socket\") pod \"426b4a4e-eb71-4780-8590-79c2a11ab95a\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " Nov 02 15:27:52.743678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743652 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-envoy\") pod \"038259bc-614d-48df-b46b-33544964ee15\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " Nov 02 15:27:52.743891 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743720 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-socket\") pod \"038259bc-614d-48df-b46b-33544964ee15\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " Nov 02 15:27:52.743891 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743760 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/038259bc-614d-48df-b46b-33544964ee15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038259bc-614d-48df-b46b-33544964ee15-istio-podinfo\") pod \"038259bc-614d-48df-b46b-33544964ee15\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " Nov 02 15:27:52.743891 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743789 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-socket" pod "25c01092-3876-411c-80f1-f1edb9e90f2b" (UID: "25c01092-3876-411c-80f1-f1edb9e90f2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.743891 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743814 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-credential-socket\") pod \"25c01092-3876-411c-80f1-f1edb9e90f2b\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " Nov 02 15:27:52.744111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743865 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-certs" pod "25c01092-3876-411c-80f1-f1edb9e90f2b" (UID: "25c01092-3876-411c-80f1-f1edb9e90f2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.744111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743905 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-socket" pod "426b4a4e-eb71-4780-8590-79c2a11ab95a" (UID: "426b4a4e-eb71-4780-8590-79c2a11ab95a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.744111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.743943 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-data\") pod \"426b4a4e-eb71-4780-8590-79c2a11ab95a\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " Nov 02 15:27:52.744111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744012 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-data\") pod \"25c01092-3876-411c-80f1-f1edb9e90f2b\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " Nov 02 15:27:52.744111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744053 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-credential-socket" pod "25c01092-3876-411c-80f1-f1edb9e90f2b" (UID: "25c01092-3876-411c-80f1-f1edb9e90f2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.744111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744057 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-kube-api-access-tjpvb\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-kube-api-access-tjpvb\") pod \"038259bc-614d-48df-b46b-33544964ee15\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " Nov 02 15:27:52.744111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744094 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-token\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-token\") pod \"25c01092-3876-411c-80f1-f1edb9e90f2b\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " Nov 02 15:27:52.744446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744138 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038259bc-614d-48df-b46b-33544964ee15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038259bc-614d-48df-b46b-33544964ee15-istiod-ca-cert\") pod \"038259bc-614d-48df-b46b-33544964ee15\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " Nov 02 15:27:52.744446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744092 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c01092-3876-411c-80f1-f1edb9e90f2b-istiod-ca-cert" pod "25c01092-3876-411c-80f1-f1edb9e90f2b" (UID: "25c01092-3876-411c-80f1-f1edb9e90f2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:52.744446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744197 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-data" pod "426b4a4e-eb71-4780-8590-79c2a11ab95a" (UID: "426b4a4e-eb71-4780-8590-79c2a11ab95a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.744446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744208 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-envoy\") pod \"25c01092-3876-411c-80f1-f1edb9e90f2b\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " Nov 02 15:27:52.744446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744254 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-data" pod "25c01092-3876-411c-80f1-f1edb9e90f2b" (UID: "25c01092-3876-411c-80f1-f1edb9e90f2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.744446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744296 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-credential-socket\") pod \"426b4a4e-eb71-4780-8590-79c2a11ab95a\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " Nov 02 15:27:52.744446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744356 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-token\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-token\") pod \"426b4a4e-eb71-4780-8590-79c2a11ab95a\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " Nov 02 15:27:52.744446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744425 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-certs\") pod \"038259bc-614d-48df-b46b-33544964ee15\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " Nov 02 15:27:52.744859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744457 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-certs\") pod \"426b4a4e-eb71-4780-8590-79c2a11ab95a\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " Nov 02 15:27:52.744859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744509 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-kube-api-access-q2qjg\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-kube-api-access-q2qjg\") pod \"25c01092-3876-411c-80f1-f1edb9e90f2b\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " Nov 02 15:27:52.744859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744552 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-envoy\") pod \"426b4a4e-eb71-4780-8590-79c2a11ab95a\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " Nov 02 15:27:52.744859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744559 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038259bc-614d-48df-b46b-33544964ee15-istiod-ca-cert" pod "038259bc-614d-48df-b46b-33544964ee15" (UID: "038259bc-614d-48df-b46b-33544964ee15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:52.744859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744588 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-credential-socket\") pod \"038259bc-614d-48df-b46b-33544964ee15\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " Nov 02 15:27:52.744859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744652 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-podinfo\") pod \"426b4a4e-eb71-4780-8590-79c2a11ab95a\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " Nov 02 15:27:52.744859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744686 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-istio-token\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-istio-token\") pod \"038259bc-614d-48df-b46b-33544964ee15\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " Nov 02 15:27:52.744859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744750 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-kube-api-access-chxt8\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-kube-api-access-chxt8\") pod \"426b4a4e-eb71-4780-8590-79c2a11ab95a\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " Nov 02 15:27:52.744859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744789 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426b4a4e-eb71-4780-8590-79c2a11ab95a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b4a4e-eb71-4780-8590-79c2a11ab95a-istiod-ca-cert\") pod \"426b4a4e-eb71-4780-8590-79c2a11ab95a\" (UID: \"426b4a4e-eb71-4780-8590-79c2a11ab95a\") " Nov 02 15:27:52.745357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744878 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-data\") pod \"038259bc-614d-48df-b46b-33544964ee15\" (UID: \"038259bc-614d-48df-b46b-33544964ee15\") " Nov 02 15:27:52.745357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744924 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-podinfo\") pod \"25c01092-3876-411c-80f1-f1edb9e90f2b\" (UID: \"25c01092-3876-411c-80f1-f1edb9e90f2b\") " Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744650 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-credential-socket" pod "426b4a4e-eb71-4780-8590-79c2a11ab95a" (UID: "426b4a4e-eb71-4780-8590-79c2a11ab95a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745779 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745800 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2hcc\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-kube-api-access-d2hcc\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745816 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745836 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c01092-3876-411c-80f1-f1edb9e90f2b-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745921 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745939 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745955 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745988 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746002 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746085 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746102 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746141 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746155 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038259bc-614d-48df-b46b-33544964ee15-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746168 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746183 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2ecad5-375c-461a-8f81-d2243d61c915-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745952 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-socket" pod "038259bc-614d-48df-b46b-33544964ee15" (UID: "038259bc-614d-48df-b46b-33544964ee15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.744804 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-certs" pod "426b4a4e-eb71-4780-8590-79c2a11ab95a" (UID: "426b4a4e-eb71-4780-8590-79c2a11ab95a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.745000 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-certs" pod "038259bc-614d-48df-b46b-33544964ee15" (UID: "038259bc-614d-48df-b46b-33544964ee15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746051 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426b4a4e-eb71-4780-8590-79c2a11ab95a-istiod-ca-cert" pod "426b4a4e-eb71-4780-8590-79c2a11ab95a" (UID: "426b4a4e-eb71-4780-8590-79c2a11ab95a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746056 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-credential-socket" pod "038259bc-614d-48df-b46b-33544964ee15" (UID: "038259bc-614d-48df-b46b-33544964ee15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.746272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.746183 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-data" pod "038259bc-614d-48df-b46b-33544964ee15" (UID: "038259bc-614d-48df-b46b-33544964ee15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.748209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.747257 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-envoy" pod "25c01092-3876-411c-80f1-f1edb9e90f2b" (UID: "25c01092-3876-411c-80f1-f1edb9e90f2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.748209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.747547 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-token" pod "25c01092-3876-411c-80f1-f1edb9e90f2b" (UID: "25c01092-3876-411c-80f1-f1edb9e90f2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.748209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.747533 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/038259bc-614d-48df-b46b-33544964ee15-istio-podinfo" pod "038259bc-614d-48df-b46b-33544964ee15" (UID: "038259bc-614d-48df-b46b-33544964ee15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:52.748209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.747829 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-envoy" pod "426b4a4e-eb71-4780-8590-79c2a11ab95a" (UID: "426b4a4e-eb71-4780-8590-79c2a11ab95a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.748209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.748042 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-envoy" pod "038259bc-614d-48df-b46b-33544964ee15" (UID: "038259bc-614d-48df-b46b-33544964ee15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.748478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.748457 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-podinfo" pod "25c01092-3876-411c-80f1-f1edb9e90f2b" (UID: "25c01092-3876-411c-80f1-f1edb9e90f2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:52.748569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.748539 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-kube-api-access-tjpvb" pod "038259bc-614d-48df-b46b-33544964ee15" (UID: "038259bc-614d-48df-b46b-33544964ee15"). InnerVolumeSpecName "kube-api-access-tjpvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.748685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.748601 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-kube-api-access-q2qjg" pod "25c01092-3876-411c-80f1-f1edb9e90f2b" (UID: "25c01092-3876-411c-80f1-f1edb9e90f2b"). InnerVolumeSpecName "kube-api-access-q2qjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.748746 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.748702 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-kube-api-access-chxt8" pod "426b4a4e-eb71-4780-8590-79c2a11ab95a" (UID: "426b4a4e-eb71-4780-8590-79c2a11ab95a"). InnerVolumeSpecName "kube-api-access-chxt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.749259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.749236 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-istio-token" pod "038259bc-614d-48df-b46b-33544964ee15" (UID: "038259bc-614d-48df-b46b-33544964ee15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.749360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.749314 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-token" pod "426b4a4e-eb71-4780-8590-79c2a11ab95a" (UID: "426b4a4e-eb71-4780-8590-79c2a11ab95a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.749648 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.749628 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-podinfo" pod "426b4a4e-eb71-4780-8590-79c2a11ab95a" (UID: "426b4a4e-eb71-4780-8590-79c2a11ab95a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:52.827771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.827684 5433 generic.go:356] "Generic (PLEG): container finished" podID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerID="c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46" exitCode=0 Nov 02 15:27:52.827771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.827752 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" Nov 02 15:27:52.828010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.827775 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" event={"ID":"25c01092-3876-411c-80f1-f1edb9e90f2b","Type":"ContainerDied","Data":"c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46"} Nov 02 15:27:52.828010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.827842 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r" event={"ID":"25c01092-3876-411c-80f1-f1edb9e90f2b","Type":"ContainerDied","Data":"c851b560f117c72f22abdbfe49a050fc4fd58d94bcd9e3c1daf4f60bb61f3c76"} Nov 02 15:27:52.828010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.827872 5433 scope.go:122] "RemoveContainer" containerID="c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46" Nov 02 15:27:52.835966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.829720 5433 generic.go:356] "Generic (PLEG): container finished" podID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerID="bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4" exitCode=0 Nov 02 15:27:52.835966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.829883 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" event={"ID":"0b2ecad5-375c-461a-8f81-d2243d61c915","Type":"ContainerDied","Data":"bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4"} Nov 02 15:27:52.835966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.829925 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" event={"ID":"0b2ecad5-375c-461a-8f81-d2243d61c915","Type":"ContainerDied","Data":"b8e5b2f9c9c6191b9da50e80e4eebb3a7b2b2c7352daea3212dd1e34876dfa04"} Nov 02 15:27:52.835966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.830036 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp" Nov 02 15:27:52.835966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.832908 5433 generic.go:356] "Generic (PLEG): container finished" podID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerID="3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3" exitCode=0 Nov 02 15:27:52.835966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.832963 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" Nov 02 15:27:52.835966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.833021 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" event={"ID":"426b4a4e-eb71-4780-8590-79c2a11ab95a","Type":"ContainerDied","Data":"3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3"} Nov 02 15:27:52.835966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.833083 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv" event={"ID":"426b4a4e-eb71-4780-8590-79c2a11ab95a","Type":"ContainerDied","Data":"d7e962490b9739c94195c802a4c497341d067f54c0e939726e5c96bdd528a6bb"} Nov 02 15:27:52.838293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.838266 5433 generic.go:356] "Generic (PLEG): container finished" podID="038259bc-614d-48df-b46b-33544964ee15" containerID="fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d" exitCode=0 Nov 02 15:27:52.838504 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.838323 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" event={"ID":"038259bc-614d-48df-b46b-33544964ee15","Type":"ContainerDied","Data":"fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d"} Nov 02 15:27:52.838504 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.838362 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" event={"ID":"038259bc-614d-48df-b46b-33544964ee15","Type":"ContainerDied","Data":"10d73549b90da5ca5d2ed67769914b3e44ba0adab403bb756ed6f845deb71eb0"} Nov 02 15:27:52.838504 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.838484 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf" Nov 02 15:27:52.847280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847257 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847286 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847303 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038259bc-614d-48df-b46b-33544964ee15-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847318 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpvb\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-kube-api-access-tjpvb\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847339 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847353 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847366 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847380 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847393 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847407 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2qjg\" (UniqueName: \"kubernetes.io/projected/25c01092-3876-411c-80f1-f1edb9e90f2b-kube-api-access-q2qjg\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847437 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847452 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847468 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b4a4e-eb71-4780-8590-79c2a11ab95a-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847481 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038259bc-614d-48df-b46b-33544964ee15-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847495 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxt8\" (UniqueName: \"kubernetes.io/projected/426b4a4e-eb71-4780-8590-79c2a11ab95a-kube-api-access-chxt8\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847509 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b4a4e-eb71-4780-8590-79c2a11ab95a-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847523 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038259bc-614d-48df-b46b-33544964ee15-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.847912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.847536 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c01092-3876-411c-80f1-f1edb9e90f2b-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.849504 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.849466 5433 scope.go:122] "RemoveContainer" containerID="c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46" Nov 02 15:27:52.850017 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:52.849826 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46\": container with ID starting with c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46 not found: ID does not exist" containerID="c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46" Nov 02 15:27:52.850017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.849877 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46"} err="failed to get container status \"c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46\": rpc error: code = NotFound desc = could not find container \"c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46\": container with ID starting with c8a2ea4df6e5abbbc0f1f18a1fcc79a6ec243ee6a7f7853f9196dd42a0a42d46 not found: ID does not exist" Nov 02 15:27:52.850017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.849902 5433 scope.go:122] "RemoveContainer" containerID="bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4" Nov 02 15:27:52.860507 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.860468 5433 scope.go:122] "RemoveContainer" containerID="bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4" Nov 02 15:27:52.860786 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:52.860766 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4\": container with ID starting with bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4 not found: ID does not exist" containerID="bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4" Nov 02 15:27:52.860844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.860793 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4"} err="failed to get container status \"bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4\": rpc error: code = NotFound desc = could not find container \"bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4\": container with ID starting with bb51f9acf3a4ff30bc69f38c3fa4b02cc29284192f60175517fea730fc2ee2f4 not found: ID does not exist" Nov 02 15:27:52.860844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.860813 5433 scope.go:122] "RemoveContainer" containerID="3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3" Nov 02 15:27:52.871191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.871171 5433 scope.go:122] "RemoveContainer" containerID="3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3" Nov 02 15:27:52.871508 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:52.871485 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3\": container with ID starting with 3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3 not found: ID does not exist" containerID="3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3" Nov 02 15:27:52.871576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.871516 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3"} err="failed to get container status \"3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3\": rpc error: code = NotFound desc = could not find container \"3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3\": container with ID starting with 3116175eaf28ed710191f3cc266fab604a4ed25e7eade3b911791ee1c959efe3 not found: ID does not exist" Nov 02 15:27:52.871576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.871535 5433 scope.go:122] "RemoveContainer" containerID="fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d" Nov 02 15:27:52.881270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.881253 5433 scope.go:122] "RemoveContainer" containerID="fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d" Nov 02 15:27:52.881813 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:27:52.881557 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d\": container with ID starting with fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d not found: ID does not exist" containerID="fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d" Nov 02 15:27:52.881813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:52.881593 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d"} err="failed to get container status \"fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d\": rpc error: code = NotFound desc = could not find container \"fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d\": container with ID starting with fac8c0e1457f03d81c1f3e985147abcd20867b971f807ee8ed36ee6778776f9d not found: ID does not exist" Nov 02 15:27:53.246867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:53.246808 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv"] Nov 02 15:27:53.252622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:53.252586 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-n69hv"] Nov 02 15:27:53.343550 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:53.343509 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp"] Nov 02 15:27:53.346374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:53.346347 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-v9tnp"] Nov 02 15:27:53.448695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:53.448660 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r"] Nov 02 15:27:53.454924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:53.454889 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-6b89r"] Nov 02 15:27:53.558444 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:53.558361 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf"] Nov 02 15:27:53.562176 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:53.562147 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-59wvf"] Nov 02 15:27:54.347442 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:54.347411 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:27:54.491654 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:54.490462 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038259bc-614d-48df-b46b-33544964ee15" path="/var/lib/kubelet/pods/038259bc-614d-48df-b46b-33544964ee15/volumes" Nov 02 15:27:54.491654 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:54.491429 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" path="/var/lib/kubelet/pods/0b2ecad5-375c-461a-8f81-d2243d61c915/volumes" Nov 02 15:27:54.493155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:54.492267 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" path="/var/lib/kubelet/pods/25c01092-3876-411c-80f1-f1edb9e90f2b/volumes" Nov 02 15:27:54.493536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:54.493457 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" path="/var/lib/kubelet/pods/426b4a4e-eb71-4780-8590-79c2a11ab95a/volumes" Nov 02 15:27:59.451940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:27:59.451898 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:02.116272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:02.116228 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:03.967999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:03.967813 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4"] Nov 02 15:28:03.968632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:03.968456 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" podUID="c00e11f9-6483-450f-ab63-8f35b4187944" containerName="maas-api" containerID="cri-o://5b128d98b929222931fd4b76f61e18a8de94b9fd7800736cd34e00a77dc025a8" gracePeriod=30 Nov 02 15:28:04.237058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:04.229543 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:04.947603 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:04.946382 5433 generic.go:356] "Generic (PLEG): container finished" podID="c00e11f9-6483-450f-ab63-8f35b4187944" containerID="5b128d98b929222931fd4b76f61e18a8de94b9fd7800736cd34e00a77dc025a8" exitCode=0 Nov 02 15:28:04.947603 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:04.946477 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" event={"ID":"c00e11f9-6483-450f-ab63-8f35b4187944","Type":"ContainerDied","Data":"5b128d98b929222931fd4b76f61e18a8de94b9fd7800736cd34e00a77dc025a8"} Nov 02 15:28:05.107689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.107663 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:28:05.335531 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.335443 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00e11f9-6483-450f-ab63-8f35b4187944-kube-api-access-l4wxj\" (UniqueName: \"kubernetes.io/projected/c00e11f9-6483-450f-ab63-8f35b4187944-kube-api-access-l4wxj\") pod \"c00e11f9-6483-450f-ab63-8f35b4187944\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " Nov 02 15:28:05.380124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.380085 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00e11f9-6483-450f-ab63-8f35b4187944-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00e11f9-6483-450f-ab63-8f35b4187944-openshift-service-ca\") pod \"c00e11f9-6483-450f-ab63-8f35b4187944\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " Nov 02 15:28:05.380474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.380446 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls\") pod \"c00e11f9-6483-450f-ab63-8f35b4187944\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " Nov 02 15:28:05.380803 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.380787 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-metrics-certs\") pod \"c00e11f9-6483-450f-ab63-8f35b4187944\" (UID: \"c00e11f9-6483-450f-ab63-8f35b4187944\") " Nov 02 15:28:05.382269 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.382243 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00e11f9-6483-450f-ab63-8f35b4187944-openshift-service-ca" pod "c00e11f9-6483-450f-ab63-8f35b4187944" (UID: "c00e11f9-6483-450f-ab63-8f35b4187944"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:28:05.456923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.456886 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls" pod "c00e11f9-6483-450f-ab63-8f35b4187944" (UID: "c00e11f9-6483-450f-ab63-8f35b4187944"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:28:05.457285 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.457090 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00e11f9-6483-450f-ab63-8f35b4187944-kube-api-access-l4wxj" pod "c00e11f9-6483-450f-ab63-8f35b4187944" (UID: "c00e11f9-6483-450f-ab63-8f35b4187944"). InnerVolumeSpecName "kube-api-access-l4wxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:28:05.468560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.468523 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-metrics-certs" pod "c00e11f9-6483-450f-ab63-8f35b4187944" (UID: "c00e11f9-6483-450f-ab63-8f35b4187944"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:28:05.485870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.485716 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4wxj\" (UniqueName: \"kubernetes.io/projected/c00e11f9-6483-450f-ab63-8f35b4187944-kube-api-access-l4wxj\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:05.485870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.485751 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00e11f9-6483-450f-ab63-8f35b4187944-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:05.485870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.485769 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:05.485870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.485784 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c00e11f9-6483-450f-ab63-8f35b4187944-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:05.957280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.957247 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" Nov 02 15:28:05.957490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.957277 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" event={"ID":"c00e11f9-6483-450f-ab63-8f35b4187944","Type":"ContainerDied","Data":"617460217d75d9d5a959687d161e941cd6cc1923e4bc5e564113f9e7dfa246eb"} Nov 02 15:28:05.957490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:05.957350 5433 scope.go:122] "RemoveContainer" containerID="5b128d98b929222931fd4b76f61e18a8de94b9fd7800736cd34e00a77dc025a8" Nov 02 15:28:06.063508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:06.063454 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:06.112893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:06.112853 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4"] Nov 02 15:28:06.159258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:06.159226 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4"] Nov 02 15:28:06.228463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:06.228371 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:06.377299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:06.377259 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:06.487130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:06.487051 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00e11f9-6483-450f-ab63-8f35b4187944" path="/var/lib/kubelet/pods/c00e11f9-6483-450f-ab63-8f35b4187944/volumes" Nov 02 15:28:08.064781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:08.064737 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0b05fd-7f45449dff-9gbj4" podUID="c00e11f9-6483-450f-ab63-8f35b4187944" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.94:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:28:08.226189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:08.226158 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl"] Nov 02 15:28:08.226404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:08.226380 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" containerID="cri-o://31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9" gracePeriod=30 Nov 02 15:28:08.233314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:08.233285 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:08.976262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:08.976228 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:09.311851 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.311764 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:09.467732 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.467697 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd"] Nov 02 15:28:09.468304 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468290 5433 cpu_manager.go:414] "containerMap: removing container" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" Nov 02 15:28:09.468345 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468306 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" Nov 02 15:28:09.468345 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468320 5433 cpu_manager.go:414] "containerMap: removing container" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" Nov 02 15:28:09.468345 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468325 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468356 5433 cpu_manager.go:414] "containerMap: removing container" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468362 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468370 5433 cpu_manager.go:414] "containerMap: removing container" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468375 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468389 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c00e11f9-6483-450f-ab63-8f35b4187944" containerName="maas-api" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468394 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00e11f9-6483-450f-ab63-8f35b4187944" containerName="maas-api" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468403 5433 cpu_manager.go:414] "containerMap: removing container" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468407 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468415 5433 cpu_manager.go:414] "containerMap: removing container" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468419 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468427 5433 cpu_manager.go:414] "containerMap: removing container" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468431 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468438 5433 cpu_manager.go:414] "containerMap: removing container" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468442 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" Nov 02 15:28:09.468445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468450 5433 cpu_manager.go:414] "containerMap: removing container" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468455 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468548 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2ecad5-375c-461a-8f81-d2243d61c915" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468556 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a2957d-3b26-49a9-862c-6ff5d83cb626" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468566 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00e11f9-6483-450f-ab63-8f35b4187944" containerName="maas-api" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468573 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cb1486-dc70-41b9-b3e8-6a25c6e9f8e1" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468579 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="426b4a4e-eb71-4780-8590-79c2a11ab95a" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468585 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="038259bc-614d-48df-b46b-33544964ee15" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468590 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f9d780-9ea7-4084-8636-b99d44a79c3f" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468596 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e411e1cf-c1c8-4d31-84fd-91a4d0167b86" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468604 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="86cca276-7d0e-4481-a1d6-d6931fdac725" containerName="istio-proxy" Nov 02 15:28:09.468861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.468609 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c01092-3876-411c-80f1-f1edb9e90f2b" containerName="istio-proxy" Nov 02 15:28:09.472358 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.472338 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.537013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.525638 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-941182-openshift-default-dockercfg-fx7hc\"" Nov 02 15:28:09.554000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.551869 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-certs\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.554000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.551948 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230df24c-62fc-42d9-a0a6-480d2e142269-istiod-ca-cert\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.554000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.552005 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230df24c-62fc-42d9-a0a6-480d2e142269-istio-podinfo\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.554000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.552064 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-envoy\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.554000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.552100 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-data\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.554000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.552204 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-credential-socket\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.554000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.552245 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-socket\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.554000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.552369 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcd75\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-kube-api-access-rcd75\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.554000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.552815 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-istio-token\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.592932 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.592842 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.694223 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd"] Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.695168 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-istio-token\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.695258 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-certs\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.695313 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230df24c-62fc-42d9-a0a6-480d2e142269-istiod-ca-cert\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.695360 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230df24c-62fc-42d9-a0a6-480d2e142269-istio-podinfo\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.695402 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-envoy\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.695436 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-data\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.695533 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-credential-socket\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.696514 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-socket\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.696624 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcd75\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-kube-api-access-rcd75\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.698616 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-certs\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.700213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.699245 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230df24c-62fc-42d9-a0a6-480d2e142269-istiod-ca-cert\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.739097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.714370 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-socket\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.739097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.714756 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-data\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.753997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.739741 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-credential-socket\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.850941 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.850302 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230df24c-62fc-42d9-a0a6-480d2e142269-istio-podinfo\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.850941 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.850808 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-istio-token\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.856014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.855875 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-envoy\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:09.859814 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:09.859752 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcd75\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-kube-api-access-rcd75\") pod \"e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:10.087638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:10.087606 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:10.264999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:10.260775 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:10.515135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:10.515104 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd"] Nov 02 15:28:10.516368 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:28:10.516333 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230df24c_62fc_42d9_a0a6_480d2e142269.slice/crio-67b585b96fc118073aa3d41a7882957b4167c243131d2292b2c90416c9d6f620 WatchSource:0}: Error finding container 67b585b96fc118073aa3d41a7882957b4167c243131d2292b2c90416c9d6f620: Status 404 returned error can't find the container with id 67b585b96fc118073aa3d41a7882957b4167c243131d2292b2c90416c9d6f620 Nov 02 15:28:10.519166 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:10.519132 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:10.519251 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:10.519201 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:10.519251 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:10.519234 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:10.993743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:10.993710 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" event={"ID":"230df24c-62fc-42d9-a0a6-480d2e142269","Type":"ContainerStarted","Data":"f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b"} Nov 02 15:28:10.993930 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:10.993764 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" event={"ID":"230df24c-62fc-42d9-a0a6-480d2e142269","Type":"ContainerStarted","Data":"67b585b96fc118073aa3d41a7882957b4167c243131d2292b2c90416c9d6f620"} Nov 02 15:28:11.013432 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:28:11.013396 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:28:11.088178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:11.088142 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:11.093632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:11.093607 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:11.115211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:11.115157 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" podStartSLOduration=2.115142345 podStartE2EDuration="2.115142345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:11.112064562 +0000 UTC m=+1794.785267756" watchObservedRunningTime="2026-11-02 15:28:11.115142345 +0000 UTC m=+1794.788345539" Nov 02 15:28:11.874103 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:11.874053 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:12.000594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:12.000561 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:12.001498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:12.001479 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:28:13.492616 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.492554 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:28:13.592740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.592706 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-credential-socket\") pod \"c8622a11-4189-477c-bd43-1cc6d12250ca\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " Nov 02 15:28:13.592916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.592810 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-data\") pod \"c8622a11-4189-477c-bd43-1cc6d12250ca\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " Nov 02 15:28:13.592916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.592907 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-socket\") pod \"c8622a11-4189-477c-bd43-1cc6d12250ca\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " Nov 02 15:28:13.593036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.592939 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-certs\") pod \"c8622a11-4189-477c-bd43-1cc6d12250ca\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " Nov 02 15:28:13.593036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.592969 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-envoy\") pod \"c8622a11-4189-477c-bd43-1cc6d12250ca\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " Nov 02 15:28:13.593126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.593063 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-credential-socket" pod "c8622a11-4189-477c-bd43-1cc6d12250ca" (UID: "c8622a11-4189-477c-bd43-1cc6d12250ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:28:13.593126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.593091 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8622a11-4189-477c-bd43-1cc6d12250ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8622a11-4189-477c-bd43-1cc6d12250ca-istiod-ca-cert\") pod \"c8622a11-4189-477c-bd43-1cc6d12250ca\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " Nov 02 15:28:13.593204 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.593115 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-data" pod "c8622a11-4189-477c-bd43-1cc6d12250ca" (UID: "c8622a11-4189-477c-bd43-1cc6d12250ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:28:13.593253 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.593193 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-podinfo\") pod \"c8622a11-4189-477c-bd43-1cc6d12250ca\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " Nov 02 15:28:13.593253 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.593237 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-certs" pod "c8622a11-4189-477c-bd43-1cc6d12250ca" (UID: "c8622a11-4189-477c-bd43-1cc6d12250ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:28:13.593330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.593253 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-socket" pod "c8622a11-4189-477c-bd43-1cc6d12250ca" (UID: "c8622a11-4189-477c-bd43-1cc6d12250ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:28:13.593330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.593267 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-token\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-token\") pod \"c8622a11-4189-477c-bd43-1cc6d12250ca\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " Nov 02 15:28:13.593330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.593320 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-kube-api-access-fb5qs\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-kube-api-access-fb5qs\") pod \"c8622a11-4189-477c-bd43-1cc6d12250ca\" (UID: \"c8622a11-4189-477c-bd43-1cc6d12250ca\") " Nov 02 15:28:13.593496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.593471 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8622a11-4189-477c-bd43-1cc6d12250ca-istiod-ca-cert" pod "c8622a11-4189-477c-bd43-1cc6d12250ca" (UID: "c8622a11-4189-477c-bd43-1cc6d12250ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:28:13.594084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.594059 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:13.594193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.594087 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:13.594193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.594102 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:13.594193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.594126 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:13.594193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.594141 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8622a11-4189-477c-bd43-1cc6d12250ca-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:13.595280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.595255 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-podinfo" pod "c8622a11-4189-477c-bd43-1cc6d12250ca" (UID: "c8622a11-4189-477c-bd43-1cc6d12250ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:28:13.595428 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.595412 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-token" pod "c8622a11-4189-477c-bd43-1cc6d12250ca" (UID: "c8622a11-4189-477c-bd43-1cc6d12250ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:28:13.595482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.595456 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-envoy" pod "c8622a11-4189-477c-bd43-1cc6d12250ca" (UID: "c8622a11-4189-477c-bd43-1cc6d12250ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:28:13.595532 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.595517 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-kube-api-access-fb5qs" pod "c8622a11-4189-477c-bd43-1cc6d12250ca" (UID: "c8622a11-4189-477c-bd43-1cc6d12250ca"). InnerVolumeSpecName "kube-api-access-fb5qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:28:13.695859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.695822 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:13.695859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.695851 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:13.695859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.695864 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb5qs\" (UniqueName: \"kubernetes.io/projected/c8622a11-4189-477c-bd43-1cc6d12250ca-kube-api-access-fb5qs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:13.696129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:13.695880 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8622a11-4189-477c-bd43-1cc6d12250ca-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:14.016579 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.016544 5433 generic.go:356] "Generic (PLEG): container finished" podID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerID="31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9" exitCode=0 Nov 02 15:28:14.016798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.016615 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" Nov 02 15:28:14.016798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.016622 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" event={"ID":"c8622a11-4189-477c-bd43-1cc6d12250ca","Type":"ContainerDied","Data":"31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9"} Nov 02 15:28:14.016798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.016716 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl" event={"ID":"c8622a11-4189-477c-bd43-1cc6d12250ca","Type":"ContainerDied","Data":"99c97e2e7b47606ae39fc8a0a44938a71e13153b9fbb9d6d66078b2f71f20249"} Nov 02 15:28:14.016798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.016748 5433 scope.go:122] "RemoveContainer" containerID="31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9" Nov 02 15:28:14.028578 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.028538 5433 scope.go:122] "RemoveContainer" containerID="066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce" Nov 02 15:28:14.038671 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.038645 5433 scope.go:122] "RemoveContainer" containerID="31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9" Nov 02 15:28:14.038946 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:28:14.038929 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9\": container with ID starting with 31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9 not found: ID does not exist" containerID="31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9" Nov 02 15:28:14.039004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.038956 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9"} err="failed to get container status \"31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9\": rpc error: code = NotFound desc = could not find container \"31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9\": container with ID starting with 31e2c79015def75087f1711b7243df28f1cd0b7f01ab1d74b042b803603895b9 not found: ID does not exist" Nov 02 15:28:14.039004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.038992 5433 scope.go:122] "RemoveContainer" containerID="066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce" Nov 02 15:28:14.039242 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:28:14.039221 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce\": container with ID starting with 066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce not found: ID does not exist" containerID="066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce" Nov 02 15:28:14.039310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.039251 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce"} err="failed to get container status \"066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce\": rpc error: code = NotFound desc = could not find container \"066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce\": container with ID starting with 066963fc77cb9930ec435937abb66e1d9a89f590a0b1d570bb5add732fd1a7ce not found: ID does not exist" Nov 02 15:28:14.049700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.049673 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:14.174596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.174555 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl"] Nov 02 15:28:14.210276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.210236 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:14.220307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.220273 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9d0b05fd-openshift-default-5888cd9d9-9wdcl"] Nov 02 15:28:14.413533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.413501 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:14.459328 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:14.459292 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" path="/var/lib/kubelet/pods/c8622a11-4189-477c-bd43-1cc6d12250ca/volumes" Nov 02 15:28:16.601038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:16.597108 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:17.289212 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.289174 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:17.491965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.491873 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:17.682833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.682802 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7"] Nov 02 15:28:17.683435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.683419 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" Nov 02 15:28:17.683435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.683435 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" Nov 02 15:28:17.683513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.683459 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" Nov 02 15:28:17.683513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.683465 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" Nov 02 15:28:17.683932 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.683568 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" Nov 02 15:28:17.687365 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.687332 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:17.689529 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.689512 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:17.799795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.799687 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-941182\"" Nov 02 15:28:17.799795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.799783 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-941182-dockercfg-9d9rh\"" Nov 02 15:28:17.862013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.800821 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7\" (UID: \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:17.862013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.836125 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5p2\" (UniqueName: \"kubernetes.io/projected/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-kube-api-access-7v5p2\") pod \"payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7\" (UID: \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:17.967004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.960060 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7\" (UID: \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:17.967004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.960204 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5p2\" (UniqueName: \"kubernetes.io/projected/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-kube-api-access-7v5p2\") pod \"payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7\" (UID: \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:17.967004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.965220 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7\" (UID: \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:17.985115 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:17.967306 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7"] Nov 02 15:28:18.037883 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.037845 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5p2\" (UniqueName: \"kubernetes.io/projected/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-kube-api-access-7v5p2\") pod \"payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7\" (UID: \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:18.042844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.042804 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp"] Nov 02 15:28:18.044236 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.044208 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8622a11-4189-477c-bd43-1cc6d12250ca" containerName="istio-proxy" Nov 02 15:28:18.048677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.048391 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp"] Nov 02 15:28:18.048677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.048489 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:18.163076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.163044 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcns6\" (UniqueName: \"kubernetes.io/projected/b577fc20-35be-4771-99e8-177ad3f37d8c-kube-api-access-zcns6\") pod \"payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp\" (UID: \"b577fc20-35be-4771-99e8-177ad3f37d8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:18.163309 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.163284 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b577fc20-35be-4771-99e8-177ad3f37d8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp\" (UID: \"b577fc20-35be-4771-99e8-177ad3f37d8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:18.264605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.264567 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcns6\" (UniqueName: \"kubernetes.io/projected/b577fc20-35be-4771-99e8-177ad3f37d8c-kube-api-access-zcns6\") pod \"payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp\" (UID: \"b577fc20-35be-4771-99e8-177ad3f37d8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:18.264812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.264708 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b577fc20-35be-4771-99e8-177ad3f37d8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp\" (UID: \"b577fc20-35be-4771-99e8-177ad3f37d8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:18.265444 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.265421 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b577fc20-35be-4771-99e8-177ad3f37d8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp\" (UID: \"b577fc20-35be-4771-99e8-177ad3f37d8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:18.304256 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.304221 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:18.324258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.324234 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcns6\" (UniqueName: \"kubernetes.io/projected/b577fc20-35be-4771-99e8-177ad3f37d8c-kube-api-access-zcns6\") pod \"payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp\" (UID: \"b577fc20-35be-4771-99e8-177ad3f37d8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:18.371913 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.371524 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:18.675021 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.674996 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7"] Nov 02 15:28:18.676362 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:28:18.676329 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb20b3ef_7cc7_455e_a06b_13c0bed51e24.slice/crio-45c1e9af2d0db98cd935c6989d9cd3dd493935b0371dd5cbf4ea667d35f4af97 WatchSource:0}: Error finding container 45c1e9af2d0db98cd935c6989d9cd3dd493935b0371dd5cbf4ea667d35f4af97: Status 404 returned error can't find the container with id 45c1e9af2d0db98cd935c6989d9cd3dd493935b0371dd5cbf4ea667d35f4af97 Nov 02 15:28:18.767205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:18.767173 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp"] Nov 02 15:28:18.768715 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:28:18.768692 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb577fc20_35be_4771_99e8_177ad3f37d8c.slice/crio-10ed1d8e9cb1e900ecac763086ca9743b88845311bec4588bdd6187cc09c56d8 WatchSource:0}: Error finding container 10ed1d8e9cb1e900ecac763086ca9743b88845311bec4588bdd6187cc09c56d8: Status 404 returned error can't find the container with id 10ed1d8e9cb1e900ecac763086ca9743b88845311bec4588bdd6187cc09c56d8 Nov 02 15:28:19.060737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:19.060649 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" event={"ID":"cb20b3ef-7cc7-455e-a06b-13c0bed51e24","Type":"ContainerStarted","Data":"7b5df584050e5ba9684d0e2ec3d99c3f7dc30a67691a2255bb7dfb1ae839e123"} Nov 02 15:28:19.060737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:19.060703 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" event={"ID":"cb20b3ef-7cc7-455e-a06b-13c0bed51e24","Type":"ContainerStarted","Data":"45c1e9af2d0db98cd935c6989d9cd3dd493935b0371dd5cbf4ea667d35f4af97"} Nov 02 15:28:19.060737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:19.060724 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:19.062326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:19.062295 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" event={"ID":"b577fc20-35be-4771-99e8-177ad3f37d8c","Type":"ContainerStarted","Data":"50461e6be60d0eee87b4c964fa608098e2d3d12498ad388a4bdf73131a58e1c1"} Nov 02 15:28:19.062448 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:19.062337 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" event={"ID":"b577fc20-35be-4771-99e8-177ad3f37d8c","Type":"ContainerStarted","Data":"10ed1d8e9cb1e900ecac763086ca9743b88845311bec4588bdd6187cc09c56d8"} Nov 02 15:28:19.062448 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:19.062371 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:19.173131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:19.173052 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" podStartSLOduration=2.173034857 podStartE2EDuration="2.173034857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:19.168825449 +0000 UTC m=+1802.842028655" watchObservedRunningTime="2026-11-02 15:28:19.173034857 +0000 UTC m=+1802.846238054" Nov 02 15:28:22.999927 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:22.999886 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:23.316747 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:23.316652 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:24.261964 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:24.261924 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:24.404953 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:24.404912 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:24.953299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:24.953266 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:26.422123 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:26.422034 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:28.396579 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:28.396547 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:30.071493 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:30.071466 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:28:30.071886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:30.071539 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:28:30.186061 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:30.185969 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" podStartSLOduration=13.185954523 podStartE2EDuration="13.185954523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:19.322454931 +0000 UTC m=+1802.995658127" watchObservedRunningTime="2026-11-02 15:28:30.185954523 +0000 UTC m=+1813.859157718" Nov 02 15:28:32.008745 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:32.008705 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:32.180822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:32.180733 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:33.997765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:33.997713 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:36.060489 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.060453 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j"] Nov 02 15:28:36.066010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.065965 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.130940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.130899 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-home\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.131114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.131064 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-model-cache\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.131192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.131165 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dst7t\" (UniqueName: \"kubernetes.io/projected/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-kube-api-access-dst7t\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.131430 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.131408 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tls-certs\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.131521 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.131453 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-dshm\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.131570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.131530 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tmp-dir\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.163318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.163292 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5aeac88e\"/\"kube-root-ca.crt\"" Nov 02 15:28:36.163318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.163298 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5aeac88e\"/\"default-dockercfg-x9xq4\"" Nov 02 15:28:36.163518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.163343 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5aeac88e\"/\"openshift-service-ca.crt\"" Nov 02 15:28:36.163518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.163447 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5aeac88e\"/\"e2e-lifecycle-model-5aeac88e-kserve-self-signed-certs\"" Nov 02 15:28:36.165532 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.165492 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:36.166599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.166576 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j"] Nov 02 15:28:36.253754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.253713 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tls-certs\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.263160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.263125 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-dshm\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.272134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.270150 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tmp-dir\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.272134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.270296 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-home\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.272134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.270482 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-model-cache\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.272134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.270525 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dst7t\" (UniqueName: \"kubernetes.io/projected/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-kube-api-access-dst7t\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.272134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.266914 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tls-certs\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.272134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.271574 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tmp-dir\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.272134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.271901 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-home\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.272134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.271920 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-model-cache\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.304000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.298304 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-dshm\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.329387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.329320 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dst7t\" (UniqueName: \"kubernetes.io/projected/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-kube-api-access-dst7t\") pod \"e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.379903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.379872 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:36.776035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:36.776006 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j"] Nov 02 15:28:36.779937 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:28:36.779903 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb5cee9_2cdf_4400_9b68_a20ebb9c5a88.slice/crio-35b747569256556eccc354313781404d25b92121bc7dd3734d87802e61653f47 WatchSource:0}: Error finding container 35b747569256556eccc354313781404d25b92121bc7dd3734d87802e61653f47: Status 404 returned error can't find the container with id 35b747569256556eccc354313781404d25b92121bc7dd3734d87802e61653f47 Nov 02 15:28:37.286429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:37.286395 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" event={"ID":"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88","Type":"ContainerStarted","Data":"6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30"} Nov 02 15:28:37.286872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:37.286439 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" event={"ID":"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88","Type":"ContainerStarted","Data":"35b747569256556eccc354313781404d25b92121bc7dd3734d87802e61653f47"} Nov 02 15:28:37.380294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:37.380265 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:37.397446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:37.397395 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" podStartSLOduration=1.397378872 podStartE2EDuration="1.397378872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:37.396302247 +0000 UTC m=+1821.069505442" watchObservedRunningTime="2026-11-02 15:28:37.397378872 +0000 UTC m=+1821.070582068" Nov 02 15:28:39.062634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:39.062601 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:41.014012 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:28:41.013937 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:28:44.265818 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:44.265777 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:44.409822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:44.409774 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:45.487497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:45.481791 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:45.871483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:45.871404 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:46.024909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.024876 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj"] Nov 02 15:28:46.029590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.029571 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.046716 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.046690 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-envoy\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.046843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.046727 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-certs\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.046843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.046754 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-token\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.046843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.046809 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-data\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.047003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.046965 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-credential-socket\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.047112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.047095 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-socket\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.047193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.047121 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-podinfo\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.047193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.047148 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dscd9\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-kube-api-access-dscd9\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.047298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.047283 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0516903c-1765-4aaf-9886-47e2ebfe0117-istiod-ca-cert\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.086421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.086390 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-ca6972-openshift-default-dockercfg-sxtqn\"" Nov 02 15:28:46.149106 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149058 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-certs\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.149106 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149102 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-token\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.149321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149133 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-data\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.149321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149221 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-credential-socket\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.149321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149298 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-socket\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.149497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149324 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-podinfo\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.149497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149355 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dscd9\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-kube-api-access-dscd9\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.149497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149443 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0516903c-1765-4aaf-9886-47e2ebfe0117-istiod-ca-cert\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.149669 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149574 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-envoy\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.149732 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.149670 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-data\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.152815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.150865 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-certs\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.152815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.151171 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0516903c-1765-4aaf-9886-47e2ebfe0117-istiod-ca-cert\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.152815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.152770 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-credential-socket\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.153282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.153257 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-podinfo\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.153407 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.153387 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-socket\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.153529 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.153513 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-envoy\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.154635 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.154612 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj"] Nov 02 15:28:46.162052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.161713 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:46.240739 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.240603 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dscd9\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-kube-api-access-dscd9\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.273813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.273744 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-token\") pod \"e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.346917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.346277 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:46.346917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.346639 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:46.382903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.381932 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:46.396091 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.396041 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:46.830351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.830319 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj"] Nov 02 15:28:46.831752 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:28:46.831724 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0516903c_1765_4aaf_9886_47e2ebfe0117.slice/crio-542ca803e91ec5d136d0dad7eee71e75949f1afcc2f3d6b5b15e60cd3d297a82 WatchSource:0}: Error finding container 542ca803e91ec5d136d0dad7eee71e75949f1afcc2f3d6b5b15e60cd3d297a82: Status 404 returned error can't find the container with id 542ca803e91ec5d136d0dad7eee71e75949f1afcc2f3d6b5b15e60cd3d297a82 Nov 02 15:28:46.834331 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.834300 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:46.834403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.834368 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:46.834403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:46.834399 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:47.077209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.077173 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:47.158703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.158672 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c"] Nov 02 15:28:47.163205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.163182 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.212861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.212833 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ff6e1f-openshift-default-dockercfg-2vs6p\"" Nov 02 15:28:47.226019 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.225949 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c"] Nov 02 15:28:47.231008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.230954 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-podinfo\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.231188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.231055 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istiod-ca-cert\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.231188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.231144 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-socket\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.231299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.231237 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-token\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.231495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.231467 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-envoy\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.231596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.231510 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwv9\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-kube-api-access-svwv9\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.231596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.231575 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-data\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.231702 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.231644 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-credential-socket\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.231748 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.231701 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-certs\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.266457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.266419 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:47.343999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.339967 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-podinfo\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.343999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.340040 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istiod-ca-cert\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.343999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.340145 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-socket\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.343999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.340191 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-token\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.343999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.340389 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-envoy\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.343999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.340422 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwv9\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-kube-api-access-svwv9\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.343999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.340453 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-data\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.343999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.340542 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-credential-socket\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.343999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.340606 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-certs\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.357653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.357254 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-certs\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.373869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.362645 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-data\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.373869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.367331 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-credential-socket\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.373869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.367487 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-socket\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.373869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.371243 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istiod-ca-cert\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.374271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.374209 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-podinfo\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.384334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.384306 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-envoy\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.417054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.416105 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:47.435553 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.435519 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwv9\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-kube-api-access-svwv9\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.436362 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.436127 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" event={"ID":"0516903c-1765-4aaf-9886-47e2ebfe0117","Type":"ContainerStarted","Data":"f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382"} Nov 02 15:28:47.436362 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.436180 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" event={"ID":"0516903c-1765-4aaf-9886-47e2ebfe0117","Type":"ContainerStarted","Data":"542ca803e91ec5d136d0dad7eee71e75949f1afcc2f3d6b5b15e60cd3d297a82"} Nov 02 15:28:47.441830 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.441798 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-token\") pod \"e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.479461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.479420 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:47.481066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.481033 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:28:47.557106 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.556935 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" podStartSLOduration=2.556915691 podStartE2EDuration="2.556915691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:47.553051543 +0000 UTC m=+1831.226254739" watchObservedRunningTime="2026-11-02 15:28:47.556915691 +0000 UTC m=+1831.230118886" Nov 02 15:28:47.616809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.616766 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:47.924712 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.924679 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c"] Nov 02 15:28:47.926574 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:28:47.926548 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd210f5_84bd_411e_8efa_8a1ee6f4587b.slice/crio-0fecb6002d9e52e740d3e2649c88c7e81c21c5fffee84ed8430953f91abdcd70 WatchSource:0}: Error finding container 0fecb6002d9e52e740d3e2649c88c7e81c21c5fffee84ed8430953f91abdcd70: Status 404 returned error can't find the container with id 0fecb6002d9e52e740d3e2649c88c7e81c21c5fffee84ed8430953f91abdcd70 Nov 02 15:28:47.929063 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.929031 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:47.929148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.929099 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:47.929148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:47.929128 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:48.366053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:48.361371 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:48.393000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:48.392295 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:48.576000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:48.573340 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" event={"ID":"9bd210f5-84bd-411e-8efa-8a1ee6f4587b","Type":"ContainerStarted","Data":"1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c"} Nov 02 15:28:48.576000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:48.573411 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" event={"ID":"9bd210f5-84bd-411e-8efa-8a1ee6f4587b","Type":"ContainerStarted","Data":"0fecb6002d9e52e740d3e2649c88c7e81c21c5fffee84ed8430953f91abdcd70"} Nov 02 15:28:48.576000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:48.573447 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:48.599005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:48.594325 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:28:48.699604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:48.699545 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" podStartSLOduration=1.699531012 podStartE2EDuration="1.699531012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:48.696360789 +0000 UTC m=+1832.369563985" watchObservedRunningTime="2026-11-02 15:28:48.699531012 +0000 UTC m=+1832.372734206" Nov 02 15:28:49.480636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:49.480601 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:49.486726 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:49.486699 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:49.577224 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:49.577191 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:49.578208 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:49.578187 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:28:50.275521 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:50.275455 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:50.581815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:50.581726 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:50.791240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:50.791203 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:51.025142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:51.025102 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:51.556605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:51.556542 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:51.747785 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:51.747753 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:51.984317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:51.984270 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:52.749908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:52.749871 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:54.001139 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.001100 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:54.250629 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.250210 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:54.432007 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.429721 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6"] Nov 02 15:28:54.440994 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.438032 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:28:54.508079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.491399 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-ca6972\"" Nov 02 15:28:54.536009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.520529 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6"] Nov 02 15:28:54.756127 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.536241 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-ca6972-dockercfg-mdtgf\"" Nov 02 15:28:54.767457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.561058 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507e533-234e-46ac-b071-5718b372f7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6\" (UID: \"c507e533-234e-46ac-b071-5718b372f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:28:54.767457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.765310 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7sr\" (UniqueName: \"kubernetes.io/projected/c507e533-234e-46ac-b071-5718b372f7b6-kube-api-access-fr7sr\") pod \"payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6\" (UID: \"c507e533-234e-46ac-b071-5718b372f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:28:54.767457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.765804 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w"] Nov 02 15:28:54.786241 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.786204 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w"] Nov 02 15:28:54.786366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.786318 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:28:54.866470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.866439 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5l72\" (UniqueName: \"kubernetes.io/projected/e920348e-e97f-4d71-901e-af22e14ff51f-kube-api-access-w5l72\") pod \"payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w\" (UID: \"e920348e-e97f-4d71-901e-af22e14ff51f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:28:54.866653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.866573 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e920348e-e97f-4d71-901e-af22e14ff51f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w\" (UID: \"e920348e-e97f-4d71-901e-af22e14ff51f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:28:54.866709 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.866668 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7sr\" (UniqueName: \"kubernetes.io/projected/c507e533-234e-46ac-b071-5718b372f7b6-kube-api-access-fr7sr\") pod \"payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6\" (UID: \"c507e533-234e-46ac-b071-5718b372f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:28:54.866942 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.866921 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507e533-234e-46ac-b071-5718b372f7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6\" (UID: \"c507e533-234e-46ac-b071-5718b372f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:28:54.867592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.867573 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507e533-234e-46ac-b071-5718b372f7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6\" (UID: \"c507e533-234e-46ac-b071-5718b372f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:28:54.928192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.928155 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7sr\" (UniqueName: \"kubernetes.io/projected/c507e533-234e-46ac-b071-5718b372f7b6-kube-api-access-fr7sr\") pod \"payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6\" (UID: \"c507e533-234e-46ac-b071-5718b372f7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:28:54.968675 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.968646 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5l72\" (UniqueName: \"kubernetes.io/projected/e920348e-e97f-4d71-901e-af22e14ff51f-kube-api-access-w5l72\") pod \"payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w\" (UID: \"e920348e-e97f-4d71-901e-af22e14ff51f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:28:54.968817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.968723 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e920348e-e97f-4d71-901e-af22e14ff51f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w\" (UID: \"e920348e-e97f-4d71-901e-af22e14ff51f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:28:54.969441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:54.969417 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e920348e-e97f-4d71-901e-af22e14ff51f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w\" (UID: \"e920348e-e97f-4d71-901e-af22e14ff51f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:28:55.026682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.026619 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5l72\" (UniqueName: \"kubernetes.io/projected/e920348e-e97f-4d71-901e-af22e14ff51f-kube-api-access-w5l72\") pod \"payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w\" (UID: \"e920348e-e97f-4d71-901e-af22e14ff51f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:28:55.061085 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.061055 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:28:55.107214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.107182 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:28:55.446107 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.441692 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6"] Nov 02 15:28:55.446107 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:28:55.445927 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc507e533_234e_46ac_b071_5718b372f7b6.slice/crio-61b70fbfcd5a65260f51aaea10675023b0dcb9dd371abb8a178edb1c366473f5 WatchSource:0}: Error finding container 61b70fbfcd5a65260f51aaea10675023b0dcb9dd371abb8a178edb1c366473f5: Status 404 returned error can't find the container with id 61b70fbfcd5a65260f51aaea10675023b0dcb9dd371abb8a178edb1c366473f5 Nov 02 15:28:55.502896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.502873 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w"] Nov 02 15:28:55.508401 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:28:55.508371 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode920348e_e97f_4d71_901e_af22e14ff51f.slice/crio-867d31ef0b28a77423730a16aba4beb272a80e039081a3fa18e6c68e77918bfa WatchSource:0}: Error finding container 867d31ef0b28a77423730a16aba4beb272a80e039081a3fa18e6c68e77918bfa: Status 404 returned error can't find the container with id 867d31ef0b28a77423730a16aba4beb272a80e039081a3fa18e6c68e77918bfa Nov 02 15:28:55.690118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.690075 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" event={"ID":"e920348e-e97f-4d71-901e-af22e14ff51f","Type":"ContainerStarted","Data":"0d4c7b30d0d0d32e1babfe8d93584c25427161245b283e0304554bcdea2334cc"} Nov 02 15:28:55.690282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.690141 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" event={"ID":"e920348e-e97f-4d71-901e-af22e14ff51f","Type":"ContainerStarted","Data":"867d31ef0b28a77423730a16aba4beb272a80e039081a3fa18e6c68e77918bfa"} Nov 02 15:28:55.690282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.690166 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:28:55.691830 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.691763 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" event={"ID":"c507e533-234e-46ac-b071-5718b372f7b6","Type":"ContainerStarted","Data":"a0291592d19be8b048a1c2f2e9bdd60b51c9ed536a6e3d7887d01f8dae8d7878"} Nov 02 15:28:55.691830 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.691818 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" event={"ID":"c507e533-234e-46ac-b071-5718b372f7b6","Type":"ContainerStarted","Data":"61b70fbfcd5a65260f51aaea10675023b0dcb9dd371abb8a178edb1c366473f5"} Nov 02 15:28:55.692001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.691848 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:28:55.801363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.801305 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" podStartSLOduration=1.801290133 podStartE2EDuration="1.801290133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:55.800537166 +0000 UTC m=+1839.473740361" watchObservedRunningTime="2026-11-02 15:28:55.801290133 +0000 UTC m=+1839.474493327" Nov 02 15:28:55.914153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:55.914098 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" podStartSLOduration=1.914082891 podStartE2EDuration="1.914082891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:55.912541757 +0000 UTC m=+1839.585744952" watchObservedRunningTime="2026-11-02 15:28:55.914082891 +0000 UTC m=+1839.587286085" Nov 02 15:28:56.417539 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:56.417451 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:58.808868 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:58.808837 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:59.069931 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.069840 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:28:59.246475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.246425 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl"] Nov 02 15:28:59.269429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.269396 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:28:59.282589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.282551 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl"] Nov 02 15:28:59.432188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.426473 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ff6e1f-dockercfg-q6gwn\"" Nov 02 15:28:59.461064 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.445774 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv"] Nov 02 15:28:59.461320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.445899 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ff6e1f\"" Nov 02 15:28:59.461406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.458465 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4c4v\" (UniqueName: \"kubernetes.io/projected/b524cbab-a209-4ee1-9b40-ed8d682af98b-kube-api-access-d4c4v\") pod \"payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl\" (UID: \"b524cbab-a209-4ee1-9b40-ed8d682af98b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:28:59.461646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.461625 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b524cbab-a209-4ee1-9b40-ed8d682af98b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl\" (UID: \"b524cbab-a209-4ee1-9b40-ed8d682af98b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:28:59.485231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.479603 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv"] Nov 02 15:28:59.485231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.479748 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:28:59.563525 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.563491 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4c4v\" (UniqueName: \"kubernetes.io/projected/b524cbab-a209-4ee1-9b40-ed8d682af98b-kube-api-access-d4c4v\") pod \"payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl\" (UID: \"b524cbab-a209-4ee1-9b40-ed8d682af98b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:28:59.563525 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.563531 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pklq\" (UniqueName: \"kubernetes.io/projected/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-kube-api-access-6pklq\") pod \"payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv\" (UID: \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:28:59.563760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.563621 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b524cbab-a209-4ee1-9b40-ed8d682af98b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl\" (UID: \"b524cbab-a209-4ee1-9b40-ed8d682af98b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:28:59.563760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.563690 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv\" (UID: \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:28:59.564430 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.564403 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b524cbab-a209-4ee1-9b40-ed8d682af98b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl\" (UID: \"b524cbab-a209-4ee1-9b40-ed8d682af98b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:28:59.621809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.621774 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4c4v\" (UniqueName: \"kubernetes.io/projected/b524cbab-a209-4ee1-9b40-ed8d682af98b-kube-api-access-d4c4v\") pod \"payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl\" (UID: \"b524cbab-a209-4ee1-9b40-ed8d682af98b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:28:59.665555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.665519 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv\" (UID: \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:28:59.665762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.665744 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pklq\" (UniqueName: \"kubernetes.io/projected/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-kube-api-access-6pklq\") pod \"payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv\" (UID: \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:28:59.666343 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.666324 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv\" (UID: \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:28:59.723348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.723276 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pklq\" (UniqueName: \"kubernetes.io/projected/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-kube-api-access-6pklq\") pod \"payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv\" (UID: \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:28:59.748101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.748074 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:28:59.810845 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:28:59.810629 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:29:00.123869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.123843 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl"] Nov 02 15:29:00.124821 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:00.124790 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb524cbab_a209_4ee1_9b40_ed8d682af98b.slice/crio-587cda4ad9d87e3a15a992e1b6554fde9b6948943baa57937031c981f6e1f7ec WatchSource:0}: Error finding container 587cda4ad9d87e3a15a992e1b6554fde9b6948943baa57937031c981f6e1f7ec: Status 404 returned error can't find the container with id 587cda4ad9d87e3a15a992e1b6554fde9b6948943baa57937031c981f6e1f7ec Nov 02 15:29:00.223185 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.223157 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv"] Nov 02 15:29:00.224109 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:00.224078 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fdf30f_90a7_47c0_b6e6_6f71caaa8d50.slice/crio-a4fa64532bd9e7186e39001a230b0bbf61e2840b3adc250d516c8dfca9c64232 WatchSource:0}: Error finding container a4fa64532bd9e7186e39001a230b0bbf61e2840b3adc250d516c8dfca9c64232: Status 404 returned error can't find the container with id a4fa64532bd9e7186e39001a230b0bbf61e2840b3adc250d516c8dfca9c64232 Nov 02 15:29:00.726712 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.726678 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" event={"ID":"b524cbab-a209-4ee1-9b40-ed8d682af98b","Type":"ContainerStarted","Data":"45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e"} Nov 02 15:29:00.726929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.726736 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" event={"ID":"b524cbab-a209-4ee1-9b40-ed8d682af98b","Type":"ContainerStarted","Data":"587cda4ad9d87e3a15a992e1b6554fde9b6948943baa57937031c981f6e1f7ec"} Nov 02 15:29:00.726929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.726765 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:29:00.728345 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.728308 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" event={"ID":"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50","Type":"ContainerStarted","Data":"885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536"} Nov 02 15:29:00.728462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.728354 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" event={"ID":"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50","Type":"ContainerStarted","Data":"a4fa64532bd9e7186e39001a230b0bbf61e2840b3adc250d516c8dfca9c64232"} Nov 02 15:29:00.728462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.728376 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:29:00.843290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.843242 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" podStartSLOduration=1.8432287170000001 podStartE2EDuration="1.843228717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:00.839049193 +0000 UTC m=+1844.512252387" watchObservedRunningTime="2026-11-02 15:29:00.843228717 +0000 UTC m=+1844.516431912" Nov 02 15:29:00.900124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.900089 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf"] Nov 02 15:29:00.900282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.900264 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" containerID="cri-o://c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3" gracePeriod=30 Nov 02 15:29:00.904480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.904456 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts"] Nov 02 15:29:00.904635 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.904617 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" containerID="cri-o://e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0" gracePeriod=30 Nov 02 15:29:00.937741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.937716 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc"] Nov 02 15:29:00.937902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.937882 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" containerID="cri-o://1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44" gracePeriod=30 Nov 02 15:29:00.940405 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.940377 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc"] Nov 02 15:29:00.940613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.940587 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" containerID="cri-o://87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b" gracePeriod=30 Nov 02 15:29:00.951582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:00.951531 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" podStartSLOduration=1.951515031 podStartE2EDuration="1.951515031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:00.948671985 +0000 UTC m=+1844.621875179" watchObservedRunningTime="2026-11-02 15:29:00.951515031 +0000 UTC m=+1844.624718226" Nov 02 15:29:01.172064 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:01.160673 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:02.090368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.090337 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v"] Nov 02 15:29:02.095033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.095005 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.152132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.152091 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-1c449c-openshift-default-dockercfg-z27gs\"" Nov 02 15:29:02.175050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.164118 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v"] Nov 02 15:29:02.188390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.188356 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:02.217715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.217675 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5n9p\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-kube-api-access-p5n9p\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.217895 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.217818 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-podinfo\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.217895 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.217871 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a39989-2f95-40cb-a48d-98addc6d6f34-istiod-ca-cert\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.218066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.218039 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-certs\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.218178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.218157 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-credential-socket\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.218249 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.218234 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-data\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.218359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.218336 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-socket\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.218434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.218417 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-token\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.218512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.218497 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-envoy\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.328665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.328630 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-socket\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.334138 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-token\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.334250 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-envoy\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.334299 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5n9p\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-kube-api-access-p5n9p\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.334386 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-podinfo\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.334437 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a39989-2f95-40cb-a48d-98addc6d6f34-istiod-ca-cert\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.334506 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-certs\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.334581 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-credential-socket\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.334658 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-data\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.335194 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-data\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.329458 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-socket\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.336179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.332901 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:02.345042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.338533 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a39989-2f95-40cb-a48d-98addc6d6f34-istiod-ca-cert\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.360998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.357161 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-certs\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.360998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.357618 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-credential-socket\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.368154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.361476 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-envoy\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.368154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.361283 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-podinfo\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.413620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.410138 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-token\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.426002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.422605 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5n9p\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-kube-api-access-p5n9p\") pod \"e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:02.522030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.519852 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:02.712790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:02.712756 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:03.145501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:03.145472 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v"] Nov 02 15:29:03.147734 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:03.147699 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a39989_2f95_40cb_a48d_98addc6d6f34.slice/crio-613ba7974c16515f4cd68a7994a54ee0416af5e0341ac3f15f10c98a38937022 WatchSource:0}: Error finding container 613ba7974c16515f4cd68a7994a54ee0416af5e0341ac3f15f10c98a38937022: Status 404 returned error can't find the container with id 613ba7974c16515f4cd68a7994a54ee0416af5e0341ac3f15f10c98a38937022 Nov 02 15:29:03.150888 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:03.150855 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:03.151034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:03.150923 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:03.151034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:03.150955 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:03.214832 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:03.214803 5433 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:29:03.214958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:03.214861 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:29:03.809304 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:03.809270 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" event={"ID":"04a39989-2f95-40cb-a48d-98addc6d6f34","Type":"ContainerStarted","Data":"3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7"} Nov 02 15:29:03.809503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:03.809328 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" event={"ID":"04a39989-2f95-40cb-a48d-98addc6d6f34","Type":"ContainerStarted","Data":"613ba7974c16515f4cd68a7994a54ee0416af5e0341ac3f15f10c98a38937022"} Nov 02 15:29:03.948347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:03.948298 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" podStartSLOduration=1.9482841899999999 podStartE2EDuration="1.94828419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:03.94466208 +0000 UTC m=+1847.617865274" watchObservedRunningTime="2026-11-02 15:29:03.94828419 +0000 UTC m=+1847.621487385" Nov 02 15:29:04.713843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:04.713807 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:04.720451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:04.720426 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:04.815837 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:04.815802 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:04.816802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:04.816780 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:29:06.266334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.266298 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:29:06.305946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.305919 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:29:06.309610 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.309587 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:29:06.313808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.313788 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:29:06.408085 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408054 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-kube-api-access-vv57p\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-kube-api-access-vv57p\") pod \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " Nov 02 15:29:06.408262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408097 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-credential-socket\") pod \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " Nov 02 15:29:06.408262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408140 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-envoy\") pod \"16240c61-804f-47e9-bfa4-57d049a4e508\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " Nov 02 15:29:06.408262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408170 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-envoy\") pod \"1d224219-f041-4188-a745-e0bf2e46f1b4\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " Nov 02 15:29:06.408262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408204 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-token\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-token\") pod \"1d224219-f041-4188-a745-e0bf2e46f1b4\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " Nov 02 15:29:06.408262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408231 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-data\") pod \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " Nov 02 15:29:06.408536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408272 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16240c61-804f-47e9-bfa4-57d049a4e508-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16240c61-804f-47e9-bfa4-57d049a4e508-istio-podinfo\") pod \"16240c61-804f-47e9-bfa4-57d049a4e508\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " Nov 02 15:29:06.408536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408336 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-credential-socket\") pod \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " Nov 02 15:29:06.408536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408375 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-podinfo\") pod \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " Nov 02 15:29:06.408536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408415 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-podinfo\") pod \"1d224219-f041-4188-a745-e0bf2e46f1b4\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " Nov 02 15:29:06.408536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408451 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-token\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-token\") pod \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " Nov 02 15:29:06.408536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408503 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-certs\") pod \"1d224219-f041-4188-a745-e0bf2e46f1b4\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " Nov 02 15:29:06.408829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408535 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d224219-f041-4188-a745-e0bf2e46f1b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d224219-f041-4188-a745-e0bf2e46f1b4-istiod-ca-cert\") pod \"1d224219-f041-4188-a745-e0bf2e46f1b4\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " Nov 02 15:29:06.408829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408577 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-socket\") pod \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " Nov 02 15:29:06.408829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408614 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-credential-socket\") pod \"1d224219-f041-4188-a745-e0bf2e46f1b4\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " Nov 02 15:29:06.408829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408663 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16240c61-804f-47e9-bfa4-57d049a4e508-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16240c61-804f-47e9-bfa4-57d049a4e508-istiod-ca-cert\") pod \"16240c61-804f-47e9-bfa4-57d049a4e508\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " Nov 02 15:29:06.408829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408693 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-socket\") pod \"16240c61-804f-47e9-bfa4-57d049a4e508\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " Nov 02 15:29:06.408829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408741 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-socket\") pod \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " Nov 02 15:29:06.409167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408899 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-certs\") pod \"16240c61-804f-47e9-bfa4-57d049a4e508\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " Nov 02 15:29:06.409167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408936 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-socket\") pod \"1d224219-f041-4188-a745-e0bf2e46f1b4\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " Nov 02 15:29:06.409167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409018 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istiod-ca-cert\") pod \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " Nov 02 15:29:06.409167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409137 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-credential-socket\") pod \"16240c61-804f-47e9-bfa4-57d049a4e508\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " Nov 02 15:29:06.409883 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.408496 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-credential-socket" pod "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" (UID: "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.410048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409894 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-socket" pod "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" (UID: "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.410048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409922 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-kube-api-access-z2z7b\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-kube-api-access-z2z7b\") pod \"16240c61-804f-47e9-bfa4-57d049a4e508\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " Nov 02 15:29:06.410048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410000 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-kube-api-access-q4r7n\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-kube-api-access-q4r7n\") pod \"1d224219-f041-4188-a745-e0bf2e46f1b4\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " Nov 02 15:29:06.410048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410031 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-token\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-token\") pod \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " Nov 02 15:29:06.410286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410093 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-data\") pod \"1d224219-f041-4188-a745-e0bf2e46f1b4\" (UID: \"1d224219-f041-4188-a745-e0bf2e46f1b4\") " Nov 02 15:29:06.410286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410136 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-istio-token\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-istio-token\") pod \"16240c61-804f-47e9-bfa4-57d049a4e508\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " Nov 02 15:29:06.410286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410189 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-podinfo\") pod \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " Nov 02 15:29:06.410286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410219 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-data\") pod \"16240c61-804f-47e9-bfa4-57d049a4e508\" (UID: \"16240c61-804f-47e9-bfa4-57d049a4e508\") " Nov 02 15:29:06.410286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410262 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-envoy\") pod \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " Nov 02 15:29:06.410286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410270 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-credential-socket" pod "16240c61-804f-47e9-bfa4-57d049a4e508" (UID: "16240c61-804f-47e9-bfa4-57d049a4e508"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.410638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410306 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-envoy\") pod \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " Nov 02 15:29:06.410638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410338 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istiod-ca-cert\") pod \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " Nov 02 15:29:06.410638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410376 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-data\") pod \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " Nov 02 15:29:06.410638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410417 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-certs\") pod \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " Nov 02 15:29:06.410638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410467 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-certs\") pod \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\" (UID: \"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c\") " Nov 02 15:29:06.410638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410495 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-kube-api-access-c4wpf\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-kube-api-access-c4wpf\") pod \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\" (UID: \"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00\") " Nov 02 15:29:06.410638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410621 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istiod-ca-cert" pod "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" (UID: "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:06.411047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.410843 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-credential-socket" pod "1d224219-f041-4188-a745-e0bf2e46f1b4" (UID: "1d224219-f041-4188-a745-e0bf2e46f1b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.411047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409453 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-credential-socket" pod "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" (UID: "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.411047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409462 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-data" pod "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" (UID: "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.411047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409492 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-socket" pod "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" (UID: "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.411047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409698 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-certs" pod "1d224219-f041-4188-a745-e0bf2e46f1b4" (UID: "1d224219-f041-4188-a745-e0bf2e46f1b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.411047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409715 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-socket" pod "1d224219-f041-4188-a745-e0bf2e46f1b4" (UID: "1d224219-f041-4188-a745-e0bf2e46f1b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.411047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.409782 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d224219-f041-4188-a745-e0bf2e46f1b4-istiod-ca-cert" pod "1d224219-f041-4188-a745-e0bf2e46f1b4" (UID: "1d224219-f041-4188-a745-e0bf2e46f1b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:06.411535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411380 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-data" pod "16240c61-804f-47e9-bfa4-57d049a4e508" (UID: "16240c61-804f-47e9-bfa4-57d049a4e508"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.411535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411396 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-socket" pod "16240c61-804f-47e9-bfa4-57d049a4e508" (UID: "16240c61-804f-47e9-bfa4-57d049a4e508"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.411535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411416 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16240c61-804f-47e9-bfa4-57d049a4e508-istiod-ca-cert" pod "16240c61-804f-47e9-bfa4-57d049a4e508" (UID: "16240c61-804f-47e9-bfa4-57d049a4e508"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:06.411699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411608 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.411699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411617 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-data" pod "1d224219-f041-4188-a745-e0bf2e46f1b4" (UID: "1d224219-f041-4188-a745-e0bf2e46f1b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.411699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411632 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.411699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411650 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.411699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411665 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.411699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411678 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.411699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411692 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d224219-f041-4188-a745-e0bf2e46f1b4-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.412074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411705 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.412074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411719 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.412074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411733 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16240c61-804f-47e9-bfa4-57d049a4e508-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.412074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411738 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istiod-ca-cert" pod "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" (UID: "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:06.412074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411747 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.412074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411761 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.412074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411774 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.412074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411789 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.412074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.411803 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.412869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.412553 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-podinfo" pod "1d224219-f041-4188-a745-e0bf2e46f1b4" (UID: "1d224219-f041-4188-a745-e0bf2e46f1b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:06.412869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.412608 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-certs" pod "16240c61-804f-47e9-bfa4-57d049a4e508" (UID: "16240c61-804f-47e9-bfa4-57d049a4e508"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.413130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.413063 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-certs" pod "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" (UID: "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.413308 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.413285 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-data" pod "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" (UID: "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.413522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.413501 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-certs" pod "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" (UID: "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.414941 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.414915 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-envoy" pod "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" (UID: "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.416080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.416053 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-envoy" pod "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" (UID: "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.416827 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.416801 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-kube-api-access-q4r7n" pod "1d224219-f041-4188-a745-e0bf2e46f1b4" (UID: "1d224219-f041-4188-a745-e0bf2e46f1b4"). InnerVolumeSpecName "kube-api-access-q4r7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.417441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.417399 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-podinfo" pod "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" (UID: "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:06.417441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.417406 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-istio-token" pod "16240c61-804f-47e9-bfa4-57d049a4e508" (UID: "16240c61-804f-47e9-bfa4-57d049a4e508"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.417441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.417426 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-token" pod "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" (UID: "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.417715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.417513 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-kube-api-access-z2z7b" pod "16240c61-804f-47e9-bfa4-57d049a4e508" (UID: "16240c61-804f-47e9-bfa4-57d049a4e508"). InnerVolumeSpecName "kube-api-access-z2z7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.417715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.417578 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-envoy" pod "1d224219-f041-4188-a745-e0bf2e46f1b4" (UID: "1d224219-f041-4188-a745-e0bf2e46f1b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.418071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.418004 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-token" pod "1d224219-f041-4188-a745-e0bf2e46f1b4" (UID: "1d224219-f041-4188-a745-e0bf2e46f1b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.418071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.418037 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16240c61-804f-47e9-bfa4-57d049a4e508-istio-podinfo" pod "16240c61-804f-47e9-bfa4-57d049a4e508" (UID: "16240c61-804f-47e9-bfa4-57d049a4e508"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:06.419547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.418343 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-kube-api-access-vv57p" pod "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" (UID: "5115eacb-6b82-4a8e-bc75-b60b9c0fd83c"). InnerVolumeSpecName "kube-api-access-vv57p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.419547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.418401 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-envoy" pod "16240c61-804f-47e9-bfa4-57d049a4e508" (UID: "16240c61-804f-47e9-bfa4-57d049a4e508"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.419547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.419101 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-kube-api-access-c4wpf" pod "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" (UID: "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00"). InnerVolumeSpecName "kube-api-access-c4wpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.419547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.419328 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-token" pod "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" (UID: "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.419726 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.419554 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-podinfo" pod "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" (UID: "4bf4f2d7-b4ae-476c-b924-7f2ad9beef00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:06.513590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513559 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513590 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16240c61-804f-47e9-bfa4-57d049a4e508-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513600 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513609 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513617 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513627 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513636 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2z7b\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-kube-api-access-z2z7b\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513645 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4r7n\" (UniqueName: \"kubernetes.io/projected/1d224219-f041-4188-a745-e0bf2e46f1b4-kube-api-access-q4r7n\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513655 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513663 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513672 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16240c61-804f-47e9-bfa4-57d049a4e508-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513680 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513689 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513697 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513705 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513714 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513722 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513731 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513739 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4wpf\" (UniqueName: \"kubernetes.io/projected/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00-kube-api-access-c4wpf\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513748 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv57p\" (UniqueName: \"kubernetes.io/projected/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c-kube-api-access-vv57p\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513758 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16240c61-804f-47e9-bfa4-57d049a4e508-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.513780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.513767 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d224219-f041-4188-a745-e0bf2e46f1b4-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.701265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.701228 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:29:06.701596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.701570 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:29:06.833526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.833436 5433 generic.go:356] "Generic (PLEG): container finished" podID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerID="c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3" exitCode=0 Nov 02 15:29:06.833526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.833486 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" Nov 02 15:29:06.833907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.833537 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" event={"ID":"1d224219-f041-4188-a745-e0bf2e46f1b4","Type":"ContainerDied","Data":"c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3"} Nov 02 15:29:06.833907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.833612 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf" event={"ID":"1d224219-f041-4188-a745-e0bf2e46f1b4","Type":"ContainerDied","Data":"aafd28db9bb587e605c8cdbdacafda671c8b02a841f77c7cabde56f27184ab52"} Nov 02 15:29:06.833907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.833645 5433 scope.go:122] "RemoveContainer" containerID="c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3" Nov 02 15:29:06.836163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.836134 5433 generic.go:356] "Generic (PLEG): container finished" podID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerID="1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44" exitCode=0 Nov 02 15:29:06.836277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.836240 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" Nov 02 15:29:06.836381 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.836348 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" event={"ID":"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c","Type":"ContainerDied","Data":"1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44"} Nov 02 15:29:06.836440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.836405 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc" event={"ID":"5115eacb-6b82-4a8e-bc75-b60b9c0fd83c","Type":"ContainerDied","Data":"b50a3043dfd9e9df9e147336e4ae0c9efd001fbc3c3f34262f5df28f49047d5f"} Nov 02 15:29:06.838272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.838249 5433 generic.go:356] "Generic (PLEG): container finished" podID="16240c61-804f-47e9-bfa4-57d049a4e508" containerID="e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0" exitCode=0 Nov 02 15:29:06.838421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.838313 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" Nov 02 15:29:06.838421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.838338 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" event={"ID":"16240c61-804f-47e9-bfa4-57d049a4e508","Type":"ContainerDied","Data":"e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0"} Nov 02 15:29:06.838421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.838376 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts" event={"ID":"16240c61-804f-47e9-bfa4-57d049a4e508","Type":"ContainerDied","Data":"ea120edc1db00ed6637d570eded5c71c7a35c2bed530bc157d141d9310e8aefc"} Nov 02 15:29:06.840308 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.840285 5433 generic.go:356] "Generic (PLEG): container finished" podID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerID="87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b" exitCode=0 Nov 02 15:29:06.840406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.840331 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" Nov 02 15:29:06.840406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.840368 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" event={"ID":"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00","Type":"ContainerDied","Data":"87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b"} Nov 02 15:29:06.840525 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.840422 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc" event={"ID":"4bf4f2d7-b4ae-476c-b924-7f2ad9beef00","Type":"ContainerDied","Data":"786e2fc082e8583771ea01ea87ff2782d2d2b073ff900c2185d66db3d5a906b9"} Nov 02 15:29:06.848651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.848624 5433 scope.go:122] "RemoveContainer" containerID="c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3" Nov 02 15:29:06.849015 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:06.848942 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3\": container with ID starting with c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3 not found: ID does not exist" containerID="c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3" Nov 02 15:29:06.849131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.849044 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3"} err="failed to get container status \"c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3\": rpc error: code = NotFound desc = could not find container \"c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3\": container with ID starting with c5b922b124bc6031b761e17adc2a52c6e5edc21b708a61ceb514a207b074b2f3 not found: ID does not exist" Nov 02 15:29:06.849131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.849067 5433 scope.go:122] "RemoveContainer" containerID="1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44" Nov 02 15:29:06.864152 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.862386 5433 scope.go:122] "RemoveContainer" containerID="1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44" Nov 02 15:29:06.864619 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:06.864592 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44\": container with ID starting with 1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44 not found: ID does not exist" containerID="1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44" Nov 02 15:29:06.864723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.864629 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44"} err="failed to get container status \"1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44\": rpc error: code = NotFound desc = could not find container \"1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44\": container with ID starting with 1ce9f952550b27d6c1b8930016b8bcebd0769cca9edb69c849e179b92d3e6b44 not found: ID does not exist" Nov 02 15:29:06.864723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.864648 5433 scope.go:122] "RemoveContainer" containerID="e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0" Nov 02 15:29:06.875284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.875269 5433 scope.go:122] "RemoveContainer" containerID="e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0" Nov 02 15:29:06.875538 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:06.875520 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0\": container with ID starting with e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0 not found: ID does not exist" containerID="e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0" Nov 02 15:29:06.875586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.875547 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0"} err="failed to get container status \"e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0\": rpc error: code = NotFound desc = could not find container \"e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0\": container with ID starting with e565148bf8ba271bb4371bb815637629139edfc506adbb0319e786ad0bb9cbb0 not found: ID does not exist" Nov 02 15:29:06.875586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.875563 5433 scope.go:122] "RemoveContainer" containerID="87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b" Nov 02 15:29:06.884699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.884684 5433 scope.go:122] "RemoveContainer" containerID="87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b" Nov 02 15:29:06.884945 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:06.884924 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b\": container with ID starting with 87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b not found: ID does not exist" containerID="87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b" Nov 02 15:29:06.885080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.884947 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b"} err="failed to get container status \"87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b\": rpc error: code = NotFound desc = could not find container \"87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b\": container with ID starting with 87a3606578f7353d91f79a6e7cb558b4b90608581c3a4941090b1bb9b854fd2b not found: ID does not exist" Nov 02 15:29:06.939267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:06.939235 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:07.159335 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.159302 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:07.246186 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.246154 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf"] Nov 02 15:29:07.255299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.255274 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-ckhlf"] Nov 02 15:29:07.387140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.387111 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:07.394436 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.394410 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc"] Nov 02 15:29:07.400147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.400121 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-p9fnc"] Nov 02 15:29:07.513006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.512888 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc"] Nov 02 15:29:07.520555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.520530 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-rrlmc"] Nov 02 15:29:07.631758 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.631721 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts"] Nov 02 15:29:07.637938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:07.637908 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-xlnts"] Nov 02 15:29:08.073339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:08.073304 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:08.457988 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:08.457948 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" path="/var/lib/kubelet/pods/16240c61-804f-47e9-bfa4-57d049a4e508/volumes" Nov 02 15:29:08.458478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:08.458461 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" path="/var/lib/kubelet/pods/1d224219-f041-4188-a745-e0bf2e46f1b4/volumes" Nov 02 15:29:08.458901 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:08.458889 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" path="/var/lib/kubelet/pods/4bf4f2d7-b4ae-476c-b924-7f2ad9beef00/volumes" Nov 02 15:29:08.459692 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:08.459674 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" path="/var/lib/kubelet/pods/5115eacb-6b82-4a8e-bc75-b60b9c0fd83c/volumes" Nov 02 15:29:10.236389 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.236349 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:10.462568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.460366 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.651928 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7"] Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.653201 5433 cpu_manager.go:414] "containerMap: removing container" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.653235 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.653262 5433 cpu_manager.go:414] "containerMap: removing container" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.653271 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.653324 5433 cpu_manager.go:414] "containerMap: removing container" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.653333 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.653377 5433 cpu_manager.go:414] "containerMap: removing container" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.653383 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" Nov 02 15:29:10.657996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.653558 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d224219-f041-4188-a745-e0bf2e46f1b4" containerName="istio-proxy" Nov 02 15:29:10.658642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.658512 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf4f2d7-b4ae-476c-b924-7f2ad9beef00" containerName="istio-proxy" Nov 02 15:29:10.658642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.658559 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5115eacb-6b82-4a8e-bc75-b60b9c0fd83c" containerName="istio-proxy" Nov 02 15:29:10.658642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.658577 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="16240c61-804f-47e9-bfa4-57d049a4e508" containerName="istio-proxy" Nov 02 15:29:10.670279 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.664946 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:10.774015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.772452 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-1c449c-dockercfg-z8nnj\"" Nov 02 15:29:10.774015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.773568 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-1c449c\"" Nov 02 15:29:10.816003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.813842 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7\" (UID: \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:10.816003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.813963 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9j8q\" (UniqueName: \"kubernetes.io/projected/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-kube-api-access-r9j8q\") pod \"payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7\" (UID: \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:10.827998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.827227 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7"] Nov 02 15:29:10.852999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.841134 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t"] Nov 02 15:29:10.852999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.850243 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:10.965155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.909991 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t"] Nov 02 15:29:10.965155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.920298 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7\" (UID: \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:10.965155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.920410 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9j8q\" (UniqueName: \"kubernetes.io/projected/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-kube-api-access-r9j8q\") pod \"payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7\" (UID: \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:10.965155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:10.923045 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7\" (UID: \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:11.019304 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.019270 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9j8q\" (UniqueName: \"kubernetes.io/projected/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-kube-api-access-r9j8q\") pod \"payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7\" (UID: \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:11.021664 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.021632 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ac0d90-7058-4c43-ba23-50c7456ca2be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t\" (UID: \"27ac0d90-7058-4c43-ba23-50c7456ca2be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:11.021806 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.021754 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhf7c\" (UniqueName: \"kubernetes.io/projected/27ac0d90-7058-4c43-ba23-50c7456ca2be-kube-api-access-hhf7c\") pod \"payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t\" (UID: \"27ac0d90-7058-4c43-ba23-50c7456ca2be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:11.031704 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:11.031673 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:29:11.123010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.122962 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhf7c\" (UniqueName: \"kubernetes.io/projected/27ac0d90-7058-4c43-ba23-50c7456ca2be-kube-api-access-hhf7c\") pod \"payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t\" (UID: \"27ac0d90-7058-4c43-ba23-50c7456ca2be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:11.123201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.123138 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ac0d90-7058-4c43-ba23-50c7456ca2be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t\" (UID: \"27ac0d90-7058-4c43-ba23-50c7456ca2be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:11.123848 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.123820 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ac0d90-7058-4c43-ba23-50c7456ca2be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t\" (UID: \"27ac0d90-7058-4c43-ba23-50c7456ca2be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:11.180298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.180256 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhf7c\" (UniqueName: \"kubernetes.io/projected/27ac0d90-7058-4c43-ba23-50c7456ca2be-kube-api-access-hhf7c\") pod \"payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t\" (UID: \"27ac0d90-7058-4c43-ba23-50c7456ca2be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:11.285350 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.285263 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:11.291835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.291810 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:11.693787 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.693763 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7"] Nov 02 15:29:11.695353 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:11.695320 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bd2643_a1b3_4c60_924b_b0d1aba3ef04.slice/crio-8c01aeb11aa4144d928cb45f8aa27355d019283ef4b5737b33cf58dc609b8392 WatchSource:0}: Error finding container 8c01aeb11aa4144d928cb45f8aa27355d019283ef4b5737b33cf58dc609b8392: Status 404 returned error can't find the container with id 8c01aeb11aa4144d928cb45f8aa27355d019283ef4b5737b33cf58dc609b8392 Nov 02 15:29:11.741338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.741305 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t"] Nov 02 15:29:11.742954 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:11.742918 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ac0d90_7058_4c43_ba23_50c7456ca2be.slice/crio-61ce90c08d74e4fe8578491364bcfb401c66b8c867f0f9d1d30e9b2297187d91 WatchSource:0}: Error finding container 61ce90c08d74e4fe8578491364bcfb401c66b8c867f0f9d1d30e9b2297187d91: Status 404 returned error can't find the container with id 61ce90c08d74e4fe8578491364bcfb401c66b8c867f0f9d1d30e9b2297187d91 Nov 02 15:29:11.790185 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.790158 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:29:11.790573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:11.790552 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:29:12.025581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.025411 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" event={"ID":"58bd2643-a1b3-4c60-924b-b0d1aba3ef04","Type":"ContainerStarted","Data":"d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676"} Nov 02 15:29:12.025581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.025484 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:12.025581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.025509 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" event={"ID":"58bd2643-a1b3-4c60-924b-b0d1aba3ef04","Type":"ContainerStarted","Data":"8c01aeb11aa4144d928cb45f8aa27355d019283ef4b5737b33cf58dc609b8392"} Nov 02 15:29:12.027395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.027362 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" event={"ID":"27ac0d90-7058-4c43-ba23-50c7456ca2be","Type":"ContainerStarted","Data":"5b2b70bbe24d1136531d77ab3c24664a2a4dc22ca8069e7e8c0a5174932c220e"} Nov 02 15:29:12.027560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.027415 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" event={"ID":"27ac0d90-7058-4c43-ba23-50c7456ca2be","Type":"ContainerStarted","Data":"61ce90c08d74e4fe8578491364bcfb401c66b8c867f0f9d1d30e9b2297187d91"} Nov 02 15:29:12.027560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.027439 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:12.157215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.157106 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" podStartSLOduration=2.157087396 podStartE2EDuration="2.157087396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:12.143171696 +0000 UTC m=+1855.816374892" watchObservedRunningTime="2026-11-02 15:29:12.157087396 +0000 UTC m=+1855.830290591" Nov 02 15:29:12.258667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.258602 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" podStartSLOduration=2.258581707 podStartE2EDuration="2.258581707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:12.255874411 +0000 UTC m=+1855.929077633" watchObservedRunningTime="2026-11-02 15:29:12.258581707 +0000 UTC m=+1855.931784903" Nov 02 15:29:12.832798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.832754 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4"] Nov 02 15:29:12.833329 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.833032 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" podUID="d117b852-c584-484a-8b70-a5a87f32a82d" containerName="maas-api" containerID="cri-o://6d279aa6174a3485479c4264c8001a497608dbd70633b4d3b8b13312cf2a3938" gracePeriod=30 Nov 02 15:29:12.981385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:12.981342 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:13.054820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.054584 5433 generic.go:356] "Generic (PLEG): container finished" podID="d117b852-c584-484a-8b70-a5a87f32a82d" containerID="6d279aa6174a3485479c4264c8001a497608dbd70633b4d3b8b13312cf2a3938" exitCode=0 Nov 02 15:29:13.056451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.056075 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" event={"ID":"d117b852-c584-484a-8b70-a5a87f32a82d","Type":"ContainerDied","Data":"6d279aa6174a3485479c4264c8001a497608dbd70633b4d3b8b13312cf2a3938"} Nov 02 15:29:13.212670 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.212644 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:29:13.306105 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.306073 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d117b852-c584-484a-8b70-a5a87f32a82d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d117b852-c584-484a-8b70-a5a87f32a82d-openshift-service-ca\") pod \"d117b852-c584-484a-8b70-a5a87f32a82d\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " Nov 02 15:29:13.306274 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.306156 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls\") pod \"d117b852-c584-484a-8b70-a5a87f32a82d\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " Nov 02 15:29:13.306274 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.306243 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d117b852-c584-484a-8b70-a5a87f32a82d-kube-api-access-zntmq\" (UniqueName: \"kubernetes.io/projected/d117b852-c584-484a-8b70-a5a87f32a82d-kube-api-access-zntmq\") pod \"d117b852-c584-484a-8b70-a5a87f32a82d\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " Nov 02 15:29:13.306380 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.306361 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-metrics-certs\") pod \"d117b852-c584-484a-8b70-a5a87f32a82d\" (UID: \"d117b852-c584-484a-8b70-a5a87f32a82d\") " Nov 02 15:29:13.306431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.306408 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d117b852-c584-484a-8b70-a5a87f32a82d-openshift-service-ca" pod "d117b852-c584-484a-8b70-a5a87f32a82d" (UID: "d117b852-c584-484a-8b70-a5a87f32a82d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:13.307024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.306998 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d117b852-c584-484a-8b70-a5a87f32a82d-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:13.308438 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.308410 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-metrics-certs" pod "d117b852-c584-484a-8b70-a5a87f32a82d" (UID: "d117b852-c584-484a-8b70-a5a87f32a82d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:29:13.308551 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.308444 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d117b852-c584-484a-8b70-a5a87f32a82d-kube-api-access-zntmq" pod "d117b852-c584-484a-8b70-a5a87f32a82d" (UID: "d117b852-c584-484a-8b70-a5a87f32a82d"). InnerVolumeSpecName "kube-api-access-zntmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:13.308551 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.308448 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls" pod "d117b852-c584-484a-8b70-a5a87f32a82d" (UID: "d117b852-c584-484a-8b70-a5a87f32a82d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:29:13.482524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.482448 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:13.482688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.482678 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d117b852-c584-484a-8b70-a5a87f32a82d-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:13.482760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:13.482751 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zntmq\" (UniqueName: \"kubernetes.io/projected/d117b852-c584-484a-8b70-a5a87f32a82d-kube-api-access-zntmq\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:14.132995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:14.132774 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" event={"ID":"d117b852-c584-484a-8b70-a5a87f32a82d","Type":"ContainerDied","Data":"32aa577636861ad77f125be5ee5f5d95261627826aa6e3eb38ba85f8c882a0da"} Nov 02 15:29:14.132995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:14.132910 5433 scope.go:122] "RemoveContainer" containerID="6d279aa6174a3485479c4264c8001a497608dbd70633b4d3b8b13312cf2a3938" Nov 02 15:29:14.133592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:14.133137 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4" Nov 02 15:29:14.345604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:14.345570 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4"] Nov 02 15:29:14.391183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:14.391100 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5b52693-78896d4696-2mfm4"] Nov 02 15:29:14.458508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:14.458474 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d117b852-c584-484a-8b70-a5a87f32a82d" path="/var/lib/kubelet/pods/d117b852-c584-484a-8b70-a5a87f32a82d/volumes" Nov 02 15:29:15.063998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.059682 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:15.178083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.178050 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5"] Nov 02 15:29:15.186020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.179950 5433 cpu_manager.go:414] "containerMap: removing container" podUID="d117b852-c584-484a-8b70-a5a87f32a82d" containerName="maas-api" Nov 02 15:29:15.186020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.181612 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d117b852-c584-484a-8b70-a5a87f32a82d" containerName="maas-api" Nov 02 15:29:15.186020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.181837 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d117b852-c584-484a-8b70-a5a87f32a82d" containerName="maas-api" Nov 02 15:29:15.220009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.218693 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.247133 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.246999 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5"] Nov 02 15:29:15.267049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.267015 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625832-5f69-4153-8ec0-fec6e8507405-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.267293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.267272 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.267383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.267363 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2t85\" (UniqueName: \"kubernetes.io/projected/a0625832-5f69-4153-8ec0-fec6e8507405-kube-api-access-k2t85\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.267539 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.267521 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.278895 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.278869 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-941182-serving-cert\"" Nov 02 15:29:15.369040 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.368931 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.369040 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.369015 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2t85\" (UniqueName: \"kubernetes.io/projected/a0625832-5f69-4153-8ec0-fec6e8507405-kube-api-access-k2t85\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.369282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.369067 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.369282 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:15.369077 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-serving-cert: secret "maas-api-e2e-shared-a-w6-941182-serving-cert" not found Nov 02 15:29:15.369282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.369124 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625832-5f69-4153-8ec0-fec6e8507405-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.369282 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:15.369145 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls podName:a0625832-5f69-4153-8ec0-fec6e8507405 nodeName:}" failed. No retries permitted until 2026-11-02 15:29:15.869129751 +0000 UTC m=+1859.542332937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls") pod "maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" (UID: "a0625832-5f69-4153-8ec0-fec6e8507405") : secret "maas-api-e2e-shared-a-w6-941182-serving-cert" not found Nov 02 15:29:15.369711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.369689 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625832-5f69-4153-8ec0-fec6e8507405-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.371536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.371516 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.478035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.476594 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:15.487322 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.487103 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2t85\" (UniqueName: \"kubernetes.io/projected/a0625832-5f69-4153-8ec0-fec6e8507405-kube-api-access-k2t85\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.705421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.705387 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:15.882293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.882257 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:15.884727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:15.884703 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:16.084869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:16.084780 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:16.144205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:16.144177 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:16.522612 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:16.522581 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5"] Nov 02 15:29:16.524141 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:16.524116 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0625832_5f69_4153_8ec0_fec6e8507405.slice/crio-2ceed37ccabfca132220dc7e56d5c1692bf10b77d718c658b1ca38c57922f449 WatchSource:0}: Error finding container 2ceed37ccabfca132220dc7e56d5c1692bf10b77d718c658b1ca38c57922f449: Status 404 returned error can't find the container with id 2ceed37ccabfca132220dc7e56d5c1692bf10b77d718c658b1ca38c57922f449 Nov 02 15:29:16.525884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:16.525863 5433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:29:17.195475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:17.195441 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" event={"ID":"a0625832-5f69-4153-8ec0-fec6e8507405","Type":"ContainerStarted","Data":"2ceed37ccabfca132220dc7e56d5c1692bf10b77d718c658b1ca38c57922f449"} Nov 02 15:29:17.276430 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:17.276397 5433 scope.go:122] "RemoveContainer" containerID="27ad989facb1859c535baa2de02ab1642ccb03c4545b32a93c8f9e072d5a2366" Nov 02 15:29:18.204636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:18.204598 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" event={"ID":"a0625832-5f69-4153-8ec0-fec6e8507405","Type":"ContainerStarted","Data":"b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0"} Nov 02 15:29:18.205190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:18.204675 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:18.335721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:18.335666 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" podStartSLOduration=2.431553632 podStartE2EDuration="3.335652636s" totalImagesPullingTime="904.099004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:18.332666158 +0000 UTC m=+1862.005869353" watchObservedRunningTime="2026-11-02 15:29:18.335652636 +0000 UTC m=+1862.008855842" Nov 02 15:29:21.378243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:21.378198 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:23.037403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:23.037368 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:23.059040 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:23.059009 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:29:23.059624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:23.059600 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:29:23.193178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:23.193143 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:23.431093 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:23.431050 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:24.216793 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:24.216758 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:29:24.490400 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:24.490313 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:25.050559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.050522 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx"] Nov 02 15:29:25.075328 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.075295 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.173863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.173814 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-data\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.174937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.174908 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-podinfo\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.175153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.175135 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-socket\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.175260 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.175245 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-certs\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.175451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.175434 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-token\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.175657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.175639 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-credential-socket\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.175794 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.175779 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd811f1-973b-4846-acda-1d3e2b2a2042-istiod-ca-cert\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.176040 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.176003 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6m7\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-kube-api-access-rn6m7\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.176301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.176282 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-envoy\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.182507 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.182461 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx"] Nov 02 15:29:25.327121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.279905 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-envoy\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.327684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.294170 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3602ca-openshift-default-dockercfg-tmw7g\"" Nov 02 15:29:25.327813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.322617 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:25.328608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.328581 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-data\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.328800 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.328142 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-data\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.335996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.329008 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-podinfo\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.370577 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.370484 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-socket\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.380071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.371045 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-socket\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.380071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.371123 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-certs\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.380071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.371233 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-token\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.380071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.371276 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-credential-socket\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.380071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.371334 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd811f1-973b-4846-acda-1d3e2b2a2042-istiod-ca-cert\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.380071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.371500 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6m7\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-kube-api-access-rn6m7\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.380071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.372312 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-certs\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.380071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.372813 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-credential-socket\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.387385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.387350 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd811f1-973b-4846-acda-1d3e2b2a2042-istiod-ca-cert\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.393734 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.393626 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-envoy\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.394806 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.394762 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-podinfo\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.436356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.436328 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-token\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.436555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.436532 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6m7\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-kube-api-access-rn6m7\") pod \"e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.447755 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.447735 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:25.629966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.629922 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:25.868351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.868312 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx"] Nov 02 15:29:25.869112 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:25.869058 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd811f1_973b_4846_acda_1d3e2b2a2042.slice/crio-a0a9c07e78ee0b96456b0ad15ce5a8d2a3426d0460944d5014051dfb6875b7bc WatchSource:0}: Error finding container a0a9c07e78ee0b96456b0ad15ce5a8d2a3426d0460944d5014051dfb6875b7bc: Status 404 returned error can't find the container with id a0a9c07e78ee0b96456b0ad15ce5a8d2a3426d0460944d5014051dfb6875b7bc Nov 02 15:29:25.872144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.872112 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:25.872227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.872179 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:25.872227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:25.872208 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:26.359076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:26.358988 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" event={"ID":"ecd811f1-973b-4846-acda-1d3e2b2a2042","Type":"ContainerStarted","Data":"b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6"} Nov 02 15:29:26.359076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:26.359039 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" event={"ID":"ecd811f1-973b-4846-acda-1d3e2b2a2042","Type":"ContainerStarted","Data":"a0a9c07e78ee0b96456b0ad15ce5a8d2a3426d0460944d5014051dfb6875b7bc"} Nov 02 15:29:26.448154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:26.448124 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:26.457671 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:26.457643 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:26.475610 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:26.475505 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" podStartSLOduration=1.475488304 podStartE2EDuration="1.475488304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:26.474729552 +0000 UTC m=+1870.147932748" watchObservedRunningTime="2026-11-02 15:29:26.475488304 +0000 UTC m=+1870.148691504" Nov 02 15:29:26.712700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:26.712662 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:27.238990 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:27.238945 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:27.372065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:27.370764 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:27.374922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:27.373097 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:29:29.414771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:29.414733 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:29.625121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:29.625055 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn"] Nov 02 15:29:29.625324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:29.625294 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" containerID="cri-o://b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7" gracePeriod=30 Nov 02 15:29:29.751832 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:29.751740 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:29.970034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:29.969996 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:32.822355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:32.822316 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:33.021622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.021579 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:33.192296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.192253 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:33.232598 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.232562 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb"] Nov 02 15:29:33.239160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.239138 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:33.293113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.293081 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3602ca\"" Nov 02 15:29:33.304030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.303999 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lz7\" (UniqueName: \"kubernetes.io/projected/80b08f02-945f-4a6c-a571-0ad175b5d2c6-kube-api-access-v7lz7\") pod \"payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb\" (UID: \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:33.304218 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.304055 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b08f02-945f-4a6c-a571-0ad175b5d2c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb\" (UID: \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:33.346359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.346324 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3602ca-dockercfg-g77k8\"" Nov 02 15:29:33.484153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.474762 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lz7\" (UniqueName: \"kubernetes.io/projected/80b08f02-945f-4a6c-a571-0ad175b5d2c6-kube-api-access-v7lz7\") pod \"payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb\" (UID: \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:33.538158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.484422 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b08f02-945f-4a6c-a571-0ad175b5d2c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb\" (UID: \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:33.538158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.485584 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b08f02-945f-4a6c-a571-0ad175b5d2c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb\" (UID: \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:33.538158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.481159 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb"] Nov 02 15:29:33.565144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.565076 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm"] Nov 02 15:29:33.582997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.573349 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lz7\" (UniqueName: \"kubernetes.io/projected/80b08f02-945f-4a6c-a571-0ad175b5d2c6-kube-api-access-v7lz7\") pod \"payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb\" (UID: \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:33.590328 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.590220 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm"] Nov 02 15:29:33.590500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.590351 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:33.693638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.693600 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3602ca-58466844f-xg4rm\" (UID: \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\") " pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:33.693805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.693715 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4b4g\" (UniqueName: \"kubernetes.io/projected/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-kube-api-access-s4b4g\") pod \"payload-processing-e2e-api-a-3602ca-58466844f-xg4rm\" (UID: \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\") " pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:33.795044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.794924 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3602ca-58466844f-xg4rm\" (UID: \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\") " pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:33.795044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.795012 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4b4g\" (UniqueName: \"kubernetes.io/projected/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-kube-api-access-s4b4g\") pod \"payload-processing-e2e-api-a-3602ca-58466844f-xg4rm\" (UID: \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\") " pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:33.795673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.795649 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3602ca-58466844f-xg4rm\" (UID: \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\") " pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:33.853287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.853251 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4b4g\" (UniqueName: \"kubernetes.io/projected/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-kube-api-access-s4b4g\") pod \"payload-processing-e2e-api-a-3602ca-58466844f-xg4rm\" (UID: \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\") " pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:33.853922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.853904 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:33.913294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:33.913263 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:34.227412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.227385 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb"] Nov 02 15:29:34.228818 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:34.228789 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b08f02_945f_4a6c_a571_0ad175b5d2c6.slice/crio-48309b78efd70bc6f9fbe65fee14b894c45844cd224a09bc70ee78ad076048fb WatchSource:0}: Error finding container 48309b78efd70bc6f9fbe65fee14b894c45844cd224a09bc70ee78ad076048fb: Status 404 returned error can't find the container with id 48309b78efd70bc6f9fbe65fee14b894c45844cd224a09bc70ee78ad076048fb Nov 02 15:29:34.318711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.318691 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm"] Nov 02 15:29:34.320272 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:34.320248 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcf7d60_a0ad_4ad9_9a3c_fdc22ff52132.slice/crio-86ea239356bb28301875c943fc92ba8b94c5c92855e8112163e38d220e077c1b WatchSource:0}: Error finding container 86ea239356bb28301875c943fc92ba8b94c5c92855e8112163e38d220e077c1b: Status 404 returned error can't find the container with id 86ea239356bb28301875c943fc92ba8b94c5c92855e8112163e38d220e077c1b Nov 02 15:29:34.516016 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.515902 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" event={"ID":"80b08f02-945f-4a6c-a571-0ad175b5d2c6","Type":"ContainerStarted","Data":"0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5"} Nov 02 15:29:34.516016 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.515967 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:34.516016 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.516008 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" event={"ID":"80b08f02-945f-4a6c-a571-0ad175b5d2c6","Type":"ContainerStarted","Data":"48309b78efd70bc6f9fbe65fee14b894c45844cd224a09bc70ee78ad076048fb"} Nov 02 15:29:34.517589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.517551 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" event={"ID":"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132","Type":"ContainerStarted","Data":"edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66"} Nov 02 15:29:34.517700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.517605 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" event={"ID":"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132","Type":"ContainerStarted","Data":"86ea239356bb28301875c943fc92ba8b94c5c92855e8112163e38d220e077c1b"} Nov 02 15:29:34.517700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.517644 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:34.628090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.628036 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" podStartSLOduration=1.628020695 podStartE2EDuration="1.628020695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:34.624715279 +0000 UTC m=+1878.297918475" watchObservedRunningTime="2026-11-02 15:29:34.628020695 +0000 UTC m=+1878.301223889" Nov 02 15:29:34.736030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:34.735957 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" podStartSLOduration=1.735941393 podStartE2EDuration="1.735941393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:34.732344768 +0000 UTC m=+1878.405547964" watchObservedRunningTime="2026-11-02 15:29:34.735941393 +0000 UTC m=+1878.409144589" Nov 02 15:29:35.021555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.021527 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:29:35.064079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.063968 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-credential-socket\") pod \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " Nov 02 15:29:35.064233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064082 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istiod-ca-cert\") pod \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " Nov 02 15:29:35.064233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064122 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-kube-api-access-6xgvz\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-kube-api-access-6xgvz\") pod \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " Nov 02 15:29:35.064233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064206 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-socket\") pod \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " Nov 02 15:29:35.064347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064277 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-envoy\") pod \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " Nov 02 15:29:35.064448 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064422 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-certs\") pod \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " Nov 02 15:29:35.064840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064668 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-token\") pod \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " Nov 02 15:29:35.065008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064748 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-credential-socket" pod "b8cb52ef-3c0a-4efc-8879-f096233b6eab" (UID: "b8cb52ef-3c0a-4efc-8879-f096233b6eab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.065008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064885 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-podinfo\") pod \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " Nov 02 15:29:35.065008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064767 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-certs" pod "b8cb52ef-3c0a-4efc-8879-f096233b6eab" (UID: "b8cb52ef-3c0a-4efc-8879-f096233b6eab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.065008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064953 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istiod-ca-cert" pod "b8cb52ef-3c0a-4efc-8879-f096233b6eab" (UID: "b8cb52ef-3c0a-4efc-8879-f096233b6eab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:35.065008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.064874 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-socket" pod "b8cb52ef-3c0a-4efc-8879-f096233b6eab" (UID: "b8cb52ef-3c0a-4efc-8879-f096233b6eab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.065315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.065028 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-data\") pod \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\" (UID: \"b8cb52ef-3c0a-4efc-8879-f096233b6eab\") " Nov 02 15:29:35.065403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.065382 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-data" pod "b8cb52ef-3c0a-4efc-8879-f096233b6eab" (UID: "b8cb52ef-3c0a-4efc-8879-f096233b6eab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.065839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.065815 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.065896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.065849 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.065896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.065866 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.065896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.065881 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.066052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.065896 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.067054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.067030 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-token" pod "b8cb52ef-3c0a-4efc-8879-f096233b6eab" (UID: "b8cb52ef-3c0a-4efc-8879-f096233b6eab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.067145 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.067054 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-podinfo" pod "b8cb52ef-3c0a-4efc-8879-f096233b6eab" (UID: "b8cb52ef-3c0a-4efc-8879-f096233b6eab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:35.067216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.067197 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-envoy" pod "b8cb52ef-3c0a-4efc-8879-f096233b6eab" (UID: "b8cb52ef-3c0a-4efc-8879-f096233b6eab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.067216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.067209 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-kube-api-access-6xgvz" pod "b8cb52ef-3c0a-4efc-8879-f096233b6eab" (UID: "b8cb52ef-3c0a-4efc-8879-f096233b6eab"). InnerVolumeSpecName "kube-api-access-6xgvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.167384 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.167339 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.167384 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.167378 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.167384 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.167388 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cb52ef-3c0a-4efc-8879-f096233b6eab-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.167706 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.167398 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xgvz\" (UniqueName: \"kubernetes.io/projected/b8cb52ef-3c0a-4efc-8879-f096233b6eab-kube-api-access-6xgvz\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.528642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.528603 5433 generic.go:356] "Generic (PLEG): container finished" podID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerID="b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7" exitCode=0 Nov 02 15:29:35.528823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.528662 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" Nov 02 15:29:35.528823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.528679 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" event={"ID":"b8cb52ef-3c0a-4efc-8879-f096233b6eab","Type":"ContainerDied","Data":"b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7"} Nov 02 15:29:35.528823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.528735 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn" event={"ID":"b8cb52ef-3c0a-4efc-8879-f096233b6eab","Type":"ContainerDied","Data":"a179241837dafcbd722a7ad2ec99fc2e2636286bf427de0053200e39af5ddf8e"} Nov 02 15:29:35.528823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.528778 5433 scope.go:122] "RemoveContainer" containerID="b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7" Nov 02 15:29:35.541275 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.541256 5433 scope.go:122] "RemoveContainer" containerID="b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7" Nov 02 15:29:35.541559 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:35.541539 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7\": container with ID starting with b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7 not found: ID does not exist" containerID="b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7" Nov 02 15:29:35.541614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.541569 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7"} err="failed to get container status \"b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7\": rpc error: code = NotFound desc = could not find container \"b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7\": container with ID starting with b1c018b46f260a6f6322d08551b6083b209dd16046205572adc9d36bed4702a7 not found: ID does not exist" Nov 02 15:29:35.684605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.684566 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn"] Nov 02 15:29:35.691621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:35.691595 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vpxgn"] Nov 02 15:29:36.459543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:36.459506 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" path="/var/lib/kubelet/pods/b8cb52ef-3c0a-4efc-8879-f096233b6eab/volumes" Nov 02 15:29:38.791697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:38.791661 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:39.836541 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:39.836500 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:40.522000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:40.515915 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:41.011931 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:41.011891 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:29:41.389649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:41.389611 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:44.334508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:44.330565 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:44.975758 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:44.975722 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:45.143688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:45.143649 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:45.302136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:45.302043 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:45.542042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:45.542007 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:29:45.542822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:45.542800 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:29:48.273921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.273873 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:48.499351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.499309 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:48.701723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.701686 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88"] Nov 02 15:29:48.702601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.702575 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" Nov 02 15:29:48.702601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.702600 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" Nov 02 15:29:48.702815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.702765 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cb52ef-3c0a-4efc-8879-f096233b6eab" containerName="istio-proxy" Nov 02 15:29:48.708057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.708034 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:29:48.762963 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.762925 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88"] Nov 02 15:29:48.791556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.790544 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2nr\" (UniqueName: \"kubernetes.io/projected/62de370c-3787-488d-b6e5-cd6d61d0ef5c-kube-api-access-qb2nr\") pod \"payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88\" (UID: \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:29:48.791556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.790668 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de370c-3787-488d-b6e5-cd6d61d0ef5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88\" (UID: \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:29:48.826616 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.826521 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0de358\"" Nov 02 15:29:48.854370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.854223 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0de358-dockercfg-kzm9j\"" Nov 02 15:29:48.893624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.892512 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2nr\" (UniqueName: \"kubernetes.io/projected/62de370c-3787-488d-b6e5-cd6d61d0ef5c-kube-api-access-qb2nr\") pod \"payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88\" (UID: \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:29:48.893624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.892651 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de370c-3787-488d-b6e5-cd6d61d0ef5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88\" (UID: \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:29:48.895211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.894626 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de370c-3787-488d-b6e5-cd6d61d0ef5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88\" (UID: \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:29:48.932405 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.932310 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg"] Nov 02 15:29:48.939471 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.939437 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:29:48.968836 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.967818 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg"] Nov 02 15:29:48.994041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.994003 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xg5\" (UniqueName: \"kubernetes.io/projected/732a4e6d-1b45-4331-a5f9-ec9fe101079a-kube-api-access-55xg5\") pod \"payload-processing-e2e-api-a-0de358-6796674449-fvtqg\" (UID: \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:29:48.994392 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:48.994366 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732a4e6d-1b45-4331-a5f9-ec9fe101079a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0de358-6796674449-fvtqg\" (UID: \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:29:49.010998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.002708 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2nr\" (UniqueName: \"kubernetes.io/projected/62de370c-3787-488d-b6e5-cd6d61d0ef5c-kube-api-access-qb2nr\") pod \"payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88\" (UID: \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:29:49.029207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.029164 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:29:49.104457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.104179 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xg5\" (UniqueName: \"kubernetes.io/projected/732a4e6d-1b45-4331-a5f9-ec9fe101079a-kube-api-access-55xg5\") pod \"payload-processing-e2e-api-a-0de358-6796674449-fvtqg\" (UID: \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:29:49.104457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.104257 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732a4e6d-1b45-4331-a5f9-ec9fe101079a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0de358-6796674449-fvtqg\" (UID: \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:29:49.105259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.105108 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732a4e6d-1b45-4331-a5f9-ec9fe101079a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0de358-6796674449-fvtqg\" (UID: \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:29:49.175918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.174931 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xg5\" (UniqueName: \"kubernetes.io/projected/732a4e6d-1b45-4331-a5f9-ec9fe101079a-kube-api-access-55xg5\") pod \"payload-processing-e2e-api-a-0de358-6796674449-fvtqg\" (UID: \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\") " pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:29:49.293477 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.293391 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:29:49.422064 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:49.421642 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62de370c_3787_488d_b6e5_cd6d61d0ef5c.slice/crio-a84947230544b87f57b909d4a7acad4e10fb8b8facde6569022e79fbb481f687 WatchSource:0}: Error finding container a84947230544b87f57b909d4a7acad4e10fb8b8facde6569022e79fbb481f687: Status 404 returned error can't find the container with id a84947230544b87f57b909d4a7acad4e10fb8b8facde6569022e79fbb481f687 Nov 02 15:29:49.423272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.423213 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88"] Nov 02 15:29:49.677838 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.677807 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg"] Nov 02 15:29:49.679892 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:49.679865 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732a4e6d_1b45_4331_a5f9_ec9fe101079a.slice/crio-5ab48e42d306e37236010e2f9ea8fb96b28632dd63e8951103a72c4e62693337 WatchSource:0}: Error finding container 5ab48e42d306e37236010e2f9ea8fb96b28632dd63e8951103a72c4e62693337: Status 404 returned error can't find the container with id 5ab48e42d306e37236010e2f9ea8fb96b28632dd63e8951103a72c4e62693337 Nov 02 15:29:49.685084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.685052 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" event={"ID":"62de370c-3787-488d-b6e5-cd6d61d0ef5c","Type":"ContainerStarted","Data":"e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343"} Nov 02 15:29:49.685253 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.685100 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" event={"ID":"62de370c-3787-488d-b6e5-cd6d61d0ef5c","Type":"ContainerStarted","Data":"a84947230544b87f57b909d4a7acad4e10fb8b8facde6569022e79fbb481f687"} Nov 02 15:29:49.685253 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.685196 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:29:49.805070 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:49.805004 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" podStartSLOduration=1.8049625009999999 podStartE2EDuration="1.804962501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:49.795810191 +0000 UTC m=+1893.469013386" watchObservedRunningTime="2026-11-02 15:29:49.804962501 +0000 UTC m=+1893.478165698" Nov 02 15:29:50.268243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:50.268204 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:50.693897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:50.693857 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" event={"ID":"732a4e6d-1b45-4331-a5f9-ec9fe101079a","Type":"ContainerStarted","Data":"d1b4bd21d743d240a8d94cde2e34b74a6eaa3d4431ff13d7bb07aab017412c28"} Nov 02 15:29:50.694326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:50.693925 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" event={"ID":"732a4e6d-1b45-4331-a5f9-ec9fe101079a","Type":"ContainerStarted","Data":"5ab48e42d306e37236010e2f9ea8fb96b28632dd63e8951103a72c4e62693337"} Nov 02 15:29:50.694326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:50.694247 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:29:50.803564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:50.803499 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" podStartSLOduration=2.803480697 podStartE2EDuration="2.803480697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:50.803081159 +0000 UTC m=+1894.476284355" watchObservedRunningTime="2026-11-02 15:29:50.803480697 +0000 UTC m=+1894.476683893" Nov 02 15:29:51.701781 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:51.701749 5433 generic.go:356] "Generic (PLEG): container finished" podID="230df24c-62fc-42d9-a0a6-480d2e142269" containerID="f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b" exitCode=0 Nov 02 15:29:51.702259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:51.701828 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" event={"ID":"230df24c-62fc-42d9-a0a6-480d2e142269","Type":"ContainerDied","Data":"f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b"} Nov 02 15:29:51.702340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:51.702325 5433 scope.go:122] "RemoveContainer" containerID="f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b" Nov 02 15:29:51.703955 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:51.703928 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:51.704079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:51.704017 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:51.704079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:51.704054 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:52.711209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:52.711176 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" event={"ID":"230df24c-62fc-42d9-a0a6-480d2e142269","Type":"ContainerStarted","Data":"f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d"} Nov 02 15:29:53.088767 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:53.088679 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:29:53.089688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:53.089655 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 15:29:53.089830 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:53.089713 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 15:29:54.088821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:54.088774 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 15:29:54.089276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:54.088850 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 15:29:54.756132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:54.756097 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:54.932006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:54.926459 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:55.088482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.088401 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 15:29:55.088482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.088463 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 15:29:55.363771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.363685 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt"] Nov 02 15:29:55.367729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.367697 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.439000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.431305 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-ca6972-serving-cert\"" Nov 02 15:29:55.480683 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.480653 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt"] Nov 02 15:29:55.499707 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.499680 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmqst\" (UniqueName: \"kubernetes.io/projected/1b75064f-9c31-48d5-b496-84582cb6c5e0-kube-api-access-lmqst\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.499857 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.499811 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.500147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.500127 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75064f-9c31-48d5-b496-84582cb6c5e0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.500189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.500167 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.575110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.575060 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:55.601561 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.601529 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75064f-9c31-48d5-b496-84582cb6c5e0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.601726 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.601573 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.601726 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.601667 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmqst\" (UniqueName: \"kubernetes.io/projected/1b75064f-9c31-48d5-b496-84582cb6c5e0-kube-api-access-lmqst\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.601844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.601810 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.601960 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:55.601940 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-serving-cert: secret "maas-api-e2e-shared-a-w1-ca6972-serving-cert" not found Nov 02 15:29:55.602061 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:29:55.602047 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls podName:1b75064f-9c31-48d5-b496-84582cb6c5e0 nodeName:}" failed. No retries permitted until 2026-11-02 15:29:56.102030225 +0000 UTC m=+1899.775233411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls") pod "maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" (UID: "1b75064f-9c31-48d5-b496-84582cb6c5e0") : secret "maas-api-e2e-shared-a-w1-ca6972-serving-cert" not found Nov 02 15:29:55.602273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.602252 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75064f-9c31-48d5-b496-84582cb6c5e0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.604066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.604048 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.693930 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.693890 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmqst\" (UniqueName: \"kubernetes.io/projected/1b75064f-9c31-48d5-b496-84582cb6c5e0-kube-api-access-lmqst\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:55.767282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.767252 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:55.930011 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:55.926215 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:29:56.088402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:56.088321 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 15:29:56.088402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:56.088373 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 15:29:56.138000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:56.137088 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:56.142280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:56.142252 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:56.281312 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:56.281233 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:56.676137 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:56.676108 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt"] Nov 02 15:29:56.677522 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:29:56.677491 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b75064f_9c31_48d5_b496_84582cb6c5e0.slice/crio-8ea39c2886cfc7eef54c7d24ccfcd827e4ea7e66ad4b3d403cc4ce075ecca677 WatchSource:0}: Error finding container 8ea39c2886cfc7eef54c7d24ccfcd827e4ea7e66ad4b3d403cc4ce075ecca677: Status 404 returned error can't find the container with id 8ea39c2886cfc7eef54c7d24ccfcd827e4ea7e66ad4b3d403cc4ce075ecca677 Nov 02 15:29:56.766398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:56.766363 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" event={"ID":"1b75064f-9c31-48d5-b496-84582cb6c5e0","Type":"ContainerStarted","Data":"8ea39c2886cfc7eef54c7d24ccfcd827e4ea7e66ad4b3d403cc4ce075ecca677"} Nov 02 15:29:57.001548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:57.001466 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:29:57.089053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:57.089014 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 15:29:57.089210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:57.089075 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 15:29:57.776764 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:57.776727 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" event={"ID":"1b75064f-9c31-48d5-b496-84582cb6c5e0","Type":"ContainerStarted","Data":"e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39"} Nov 02 15:29:57.777276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:57.776928 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:29:57.898492 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:57.898415 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" podStartSLOduration=2.016731545 podStartE2EDuration="2.898396404s" totalImagesPullingTime="881.664859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:57.895420404 +0000 UTC m=+1901.568623599" watchObservedRunningTime="2026-11-02 15:29:57.898396404 +0000 UTC m=+1901.571599599" Nov 02 15:29:58.088114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:58.088066 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 15:29:58.088398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:58.088363 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 15:29:59.093765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:59.093733 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:29:59.094841 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:29:59.094818 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:30:00.697411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:00.697382 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:30:01.270782 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:01.270733 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:01.438509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:01.438476 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:01.636929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:01.636892 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:01.705788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:01.705759 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:30:03.789343 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:03.789304 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:30:07.349872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:07.349827 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:11.012468 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:30:11.012428 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:30:12.518700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:12.518667 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:17.658461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:17.658425 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:22.666999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.666757 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6"] Nov 02 15:30:22.672165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.672142 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.696609 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.696585 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6"] Nov 02 15:30:22.768000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.755961 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-1c449c-serving-cert\"" Nov 02 15:30:22.802931 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.802891 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.803332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.803306 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfwb\" (UniqueName: \"kubernetes.io/projected/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-kube-api-access-2lfwb\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.803585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.803561 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.803798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.803777 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.875194 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.875161 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:22.907041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.905408 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.907041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.905581 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfwb\" (UniqueName: \"kubernetes.io/projected/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-kube-api-access-2lfwb\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.907041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.905656 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.907041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.905795 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.907041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.906825 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:22.922758 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:30:22.908853 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-serving-cert: secret "maas-api-e2e-shared-a-w4-1c449c-serving-cert" not found Nov 02 15:30:22.922758 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:30:22.908934 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls podName:e2ef7d95-c50b-4f8c-8a89-1b8f573041d4 nodeName:}" failed. No retries permitted until 2026-11-02 15:30:23.408912481 +0000 UTC m=+1927.082115655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls") pod "maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" (UID: "e2ef7d95-c50b-4f8c-8a89-1b8f573041d4") : secret "maas-api-e2e-shared-a-w4-1c449c-serving-cert" not found Nov 02 15:30:22.953002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:22.951715 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:23.011996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:23.011557 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfwb\" (UniqueName: \"kubernetes.io/projected/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-kube-api-access-2lfwb\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:23.066772 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:23.066737 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:23.246052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:23.245843 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:23.444738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:23.444210 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:23.469013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:23.468942 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:23.585717 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:23.585631 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:24.043999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:24.042029 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6"] Nov 02 15:30:24.093812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:24.093770 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" event={"ID":"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4","Type":"ContainerStarted","Data":"de9a6a8ea58638de75d23c27014fcee51b59eb9cb7d298f2b0188bea598bdda9"} Nov 02 15:30:26.111675 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:26.111640 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" event={"ID":"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4","Type":"ContainerStarted","Data":"ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4"} Nov 02 15:30:26.112125 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:26.112076 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:26.242999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:26.233482 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" podStartSLOduration=3.228107049 podStartE2EDuration="4.233463598s" totalImagesPullingTime="1.005356549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:26.224584923 +0000 UTC m=+1929.897788119" watchObservedRunningTime="2026-11-02 15:30:26.233463598 +0000 UTC m=+1929.906666794" Nov 02 15:30:27.056716 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:27.056680 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:27.203071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:27.203034 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:28.555275 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:28.555234 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:31.159708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:31.159669 5433 generic.go:356] "Generic (PLEG): container finished" podID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerID="1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c" exitCode=0 Nov 02 15:30:31.160663 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:31.159743 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" event={"ID":"9bd210f5-84bd-411e-8efa-8a1ee6f4587b","Type":"ContainerDied","Data":"1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c"} Nov 02 15:30:31.160663 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:31.160345 5433 scope.go:122] "RemoveContainer" containerID="1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c" Nov 02 15:30:31.162178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:31.162147 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:30:31.162292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:31.162235 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:30:31.162292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:31.162286 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:30:32.144031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.143969 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:30:32.168259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.168230 5433 generic.go:356] "Generic (PLEG): container finished" podID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerID="f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382" exitCode=0 Nov 02 15:30:32.168709 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.168306 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" event={"ID":"0516903c-1765-4aaf-9886-47e2ebfe0117","Type":"ContainerDied","Data":"f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382"} Nov 02 15:30:32.168779 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.168717 5433 scope.go:122] "RemoveContainer" containerID="f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382" Nov 02 15:30:32.170467 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.170436 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:30:32.170570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.170527 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:30:32.170615 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.170576 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:30:32.170681 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.170655 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" event={"ID":"9bd210f5-84bd-411e-8efa-8a1ee6f4587b","Type":"ContainerStarted","Data":"fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c"} Nov 02 15:30:32.479742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.479658 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:30:32.480835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.480804 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 15:30:32.480969 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:32.480867 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 15:30:33.181847 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:33.181814 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" event={"ID":"0516903c-1765-4aaf-9886-47e2ebfe0117","Type":"ContainerStarted","Data":"cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a"} Nov 02 15:30:33.365438 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:33.365397 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:30:33.367442 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:33.367397 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 15:30:33.367616 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:33.367506 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 15:30:33.480032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:33.479942 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 15:30:33.480032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:33.480004 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 15:30:33.573196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:33.573157 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:30:33.693816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:33.693771 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:34.359997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:34.359369 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 15:30:34.359997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:34.359450 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 15:30:34.480042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:34.480005 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 15:30:34.480255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:34.480063 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 15:30:34.577200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:34.577164 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:30:35.359472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:35.359426 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 15:30:35.359798 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:35.359765 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 15:30:35.480366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:35.480327 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 15:30:35.480783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:35.480395 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 15:30:36.357996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:36.357351 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 15:30:36.357996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:36.357418 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 15:30:36.480404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:36.480376 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 15:30:36.480870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:36.480846 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 15:30:37.356416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:37.356375 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 15:30:37.356716 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:37.356684 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 15:30:37.480279 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:37.480249 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 15:30:37.480504 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:37.480483 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 15:30:38.356488 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:38.356450 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 02 15:30:38.356719 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:38.356519 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 02 15:30:38.583918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:38.583880 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:30:38.611072 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:38.600351 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:30:39.363967 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:39.363937 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:30:39.365025 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:39.365000 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:30:39.370571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:39.370546 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:41.011125 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:30:41.011092 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:30:42.382304 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:42.382269 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:42.841890 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:42.841802 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:43.907804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:43.907759 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:44.089860 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:44.089824 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:44.982216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:44.982167 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:47.615897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:47.615859 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:47.815678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:47.815640 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:47.969191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:47.969152 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:50.117479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.117448 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6"] Nov 02 15:30:50.121388 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.121368 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.213218 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.213187 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.213466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.213443 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66bc869-5f79-4322-873c-c7aea554c7b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.213538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.213495 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-metrics-certs\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.213583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.213561 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75fh2\" (UniqueName: \"kubernetes.io/projected/b66bc869-5f79-4322-873c-c7aea554c7b7-kube-api-access-75fh2\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.216145 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.216124 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3602ca-serving-cert\"" Nov 02 15:30:50.219575 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.219554 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6"] Nov 02 15:30:50.315283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.315254 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66bc869-5f79-4322-873c-c7aea554c7b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.315485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.315297 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-metrics-certs\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.315485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.315316 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75fh2\" (UniqueName: \"kubernetes.io/projected/b66bc869-5f79-4322-873c-c7aea554c7b7-kube-api-access-75fh2\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.315485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.315397 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.315674 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:30:50.315529 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-serving-cert: secret "maas-api-e2e-api-a-3602ca-serving-cert" not found Nov 02 15:30:50.315674 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:30:50.315589 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls podName:b66bc869-5f79-4322-873c-c7aea554c7b7 nodeName:}" failed. No retries permitted until 2026-11-02 15:30:50.815571288 +0000 UTC m=+1954.488774473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls") pod "maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" (UID: "b66bc869-5f79-4322-873c-c7aea554c7b7") : secret "maas-api-e2e-api-a-3602ca-serving-cert" not found Nov 02 15:30:50.315914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.315890 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66bc869-5f79-4322-873c-c7aea554c7b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.317888 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.317862 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-metrics-certs\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.320135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.320108 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:50.373470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.373387 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75fh2\" (UniqueName: \"kubernetes.io/projected/b66bc869-5f79-4322-873c-c7aea554c7b7-kube-api-access-75fh2\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.534867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.534801 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:50.684296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.684253 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:50.832629 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.832599 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:50.839374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.839345 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.841802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.841779 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls\") pod \"maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:50.998848 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:50.996515 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:30:51.042697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.038293 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:51.233077 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.232962 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg"] Nov 02 15:30:51.239090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.239066 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:30:51.289180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.289091 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-fb80ca\"" Nov 02 15:30:51.289736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.289700 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-fb80ca-dockercfg-pkvhl\"" Nov 02 15:30:51.294433 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.294407 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg"] Nov 02 15:30:51.403005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.401362 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l99kj\" (UniqueName: \"kubernetes.io/projected/19a3bbeb-074e-48eb-99e3-4186eee1155b-kube-api-access-l99kj\") pod \"payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg\" (UID: \"19a3bbeb-074e-48eb-99e3-4186eee1155b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:30:51.403005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.401679 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a3bbeb-074e-48eb-99e3-4186eee1155b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg\" (UID: \"19a3bbeb-074e-48eb-99e3-4186eee1155b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:30:51.543347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.540856 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a3bbeb-074e-48eb-99e3-4186eee1155b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg\" (UID: \"19a3bbeb-074e-48eb-99e3-4186eee1155b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:30:51.543347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.541130 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99kj\" (UniqueName: \"kubernetes.io/projected/19a3bbeb-074e-48eb-99e3-4186eee1155b-kube-api-access-l99kj\") pod \"payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg\" (UID: \"19a3bbeb-074e-48eb-99e3-4186eee1155b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:30:51.543347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.541293 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6"] Nov 02 15:30:51.543347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.542204 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a3bbeb-074e-48eb-99e3-4186eee1155b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg\" (UID: \"19a3bbeb-074e-48eb-99e3-4186eee1155b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:30:51.599497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.599466 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99kj\" (UniqueName: \"kubernetes.io/projected/19a3bbeb-074e-48eb-99e3-4186eee1155b-kube-api-access-l99kj\") pod \"payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg\" (UID: \"19a3bbeb-074e-48eb-99e3-4186eee1155b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:30:51.854413 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:51.854302 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:30:52.226418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:52.226300 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg"] Nov 02 15:30:52.228131 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:30:52.228100 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a3bbeb_074e_48eb_99e3_4186eee1155b.slice/crio-9a0c1bdd4c12cbc8ff04c2753505d6a4576505d61ed5f1e2e99d1d310ae0ba4e WatchSource:0}: Error finding container 9a0c1bdd4c12cbc8ff04c2753505d6a4576505d61ed5f1e2e99d1d310ae0ba4e: Status 404 returned error can't find the container with id 9a0c1bdd4c12cbc8ff04c2753505d6a4576505d61ed5f1e2e99d1d310ae0ba4e Nov 02 15:30:52.384264 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:52.384232 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" event={"ID":"b66bc869-5f79-4322-873c-c7aea554c7b7","Type":"ContainerStarted","Data":"18506248f3f5b7f1bd25efb4c3ac0784f8f2ea3be8ecb10be79b78de8b470289"} Nov 02 15:30:52.385843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:52.385818 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" event={"ID":"19a3bbeb-074e-48eb-99e3-4186eee1155b","Type":"ContainerStarted","Data":"81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57"} Nov 02 15:30:52.385916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:52.385870 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:30:52.385916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:52.385889 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" event={"ID":"19a3bbeb-074e-48eb-99e3-4186eee1155b","Type":"ContainerStarted","Data":"9a0c1bdd4c12cbc8ff04c2753505d6a4576505d61ed5f1e2e99d1d310ae0ba4e"} Nov 02 15:30:52.497048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:52.496952 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" podStartSLOduration=1.496937185 podStartE2EDuration="1.496937185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:52.494189567 +0000 UTC m=+1956.167392763" watchObservedRunningTime="2026-11-02 15:30:52.496937185 +0000 UTC m=+1956.170140380" Nov 02 15:30:53.394231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:53.394188 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" event={"ID":"b66bc869-5f79-4322-873c-c7aea554c7b7","Type":"ContainerStarted","Data":"ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d"} Nov 02 15:30:53.505419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:53.505346 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" podStartSLOduration=2.646675019 podStartE2EDuration="3.505327256s" totalImagesPullingTime="858.652237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:53.504729535 +0000 UTC m=+1957.177932732" watchObservedRunningTime="2026-11-02 15:30:53.505327256 +0000 UTC m=+1957.178530463" Nov 02 15:30:54.401403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:54.401366 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:30:56.205440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:30:56.205407 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:00.413754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:00.413722 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:31:01.367518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:01.367483 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:03.398340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:03.398307 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:31:06.456459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:06.456423 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:06.629900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:06.629854 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:06.787465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:06.787362 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:11.016186 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:31:11.016131 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:31:11.700195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:11.700124 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:12.658602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:12.658520 5433 generic.go:356] "Generic (PLEG): container finished" podID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerID="3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7" exitCode=0 Nov 02 15:31:12.659044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:12.658592 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" event={"ID":"04a39989-2f95-40cb-a48d-98addc6d6f34","Type":"ContainerDied","Data":"3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7"} Nov 02 15:31:12.659120 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:12.659064 5433 scope.go:122] "RemoveContainer" containerID="3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7" Nov 02 15:31:12.661029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:12.661003 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:12.661125 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:12.661075 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:12.661125 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:12.661108 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:13.668846 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:13.668805 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" event={"ID":"04a39989-2f95-40cb-a48d-98addc6d6f34","Type":"ContainerStarted","Data":"c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6"} Nov 02 15:31:13.713720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:13.713689 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:31:13.713902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:13.713844 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:31:13.713996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:13.713907 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 15:31:14.713917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:14.713878 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:31:14.714347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:14.713945 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 15:31:15.714050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:15.714017 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:31:15.714483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:15.714071 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 15:31:16.713425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:16.713395 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:31:16.713627 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:16.713447 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 15:31:16.905930 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:16.905890 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:17.713803 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:17.713763 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:31:17.714047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:17.713866 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 15:31:18.713461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:18.713421 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:31:18.713889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:18.713481 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 15:31:19.718564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:19.718535 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:31:19.719057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:19.718998 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:31:19.719737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:19.719719 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:31:22.061608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:22.061573 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:27.210609 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:27.210528 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:28.123410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:28.123375 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:29.253116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:29.253086 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:29.743466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:29.743416 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:30.379856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:30.379824 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:30.552950 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:30.552892 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:32.838325 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:32.838290 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:35.192121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:35.192081 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:35.363363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:35.363328 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:35.521474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:35.521389 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:35.668559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:35.668525 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:36.915126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:36.915094 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:37.067013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:37.066955 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:37.224955 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:37.224845 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:38.372816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.372785 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:38.524074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.524021 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:38.684054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.683970 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:38.815905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.815870 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts"] Nov 02 15:31:38.820042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.820012 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:38.868770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.868738 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vrsv\" (UniqueName: \"kubernetes.io/projected/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-kube-api-access-6vrsv\") pod \"payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts\" (UID: \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:38.868928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.868865 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts\" (UID: \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:38.874884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.874857 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts"] Nov 02 15:31:38.917396 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.917363 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-adf096\"" Nov 02 15:31:38.917576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.917414 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-adf096-dockercfg-dcp8l\"" Nov 02 15:31:38.969947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.969864 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts\" (UID: \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:38.970181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.970160 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vrsv\" (UniqueName: \"kubernetes.io/projected/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-kube-api-access-6vrsv\") pod \"payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts\" (UID: \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:38.970704 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:38.970684 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts\" (UID: \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:39.061361 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:39.061328 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vrsv\" (UniqueName: \"kubernetes.io/projected/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-kube-api-access-6vrsv\") pod \"payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts\" (UID: \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:39.203430 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:39.202889 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:39.623810 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:39.623770 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts"] Nov 02 15:31:39.624262 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:31:39.624176 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92810d1_ef00_4c5a_98cc_ea7eb2d7708a.slice/crio-abfef5ee8f53f784032851210d00b5dd789a6b0d3932b647edc49c22ba383cd5 WatchSource:0}: Error finding container abfef5ee8f53f784032851210d00b5dd789a6b0d3932b647edc49c22ba383cd5: Status 404 returned error can't find the container with id abfef5ee8f53f784032851210d00b5dd789a6b0d3932b647edc49c22ba383cd5 Nov 02 15:31:39.860171 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:39.860133 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" event={"ID":"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a","Type":"ContainerStarted","Data":"0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5"} Nov 02 15:31:39.860360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:39.860192 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" event={"ID":"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a","Type":"ContainerStarted","Data":"abfef5ee8f53f784032851210d00b5dd789a6b0d3932b647edc49c22ba383cd5"} Nov 02 15:31:39.860360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:39.860217 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:39.973713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:39.973660 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" podStartSLOduration=1.973645929 podStartE2EDuration="1.973645929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:39.969620007 +0000 UTC m=+2003.642823205" watchObservedRunningTime="2026-11-02 15:31:39.973645929 +0000 UTC m=+2003.646849161" Nov 02 15:31:41.012027 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:31:41.011991 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:31:44.037339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:44.037303 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:49.681749 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:49.681718 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:50.874623 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:50.874593 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:31:50.949284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:50.949252 5433 generic.go:356] "Generic (PLEG): container finished" podID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerID="b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6" exitCode=0 Nov 02 15:31:50.949454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:50.949324 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" event={"ID":"ecd811f1-973b-4846-acda-1d3e2b2a2042","Type":"ContainerDied","Data":"b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6"} Nov 02 15:31:50.949768 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:50.949754 5433 scope.go:122] "RemoveContainer" containerID="b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6" Nov 02 15:31:50.951497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:50.951467 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:50.951608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:50.951544 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:50.951608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:50.951582 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:51.958648 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:51.958612 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" event={"ID":"ecd811f1-973b-4846-acda-1d3e2b2a2042","Type":"ContainerStarted","Data":"9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47"} Nov 02 15:31:52.448364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:52.448319 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:31:52.448593 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:52.448567 5433 patch_prober.go:29] interesting pod/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 15:31:52.448647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:52.448632 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 15:31:53.448291 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:53.448259 5433 patch_prober.go:29] interesting pod/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 15:31:53.448793 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:53.448333 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 15:31:54.448461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:54.448412 5433 patch_prober.go:29] interesting pod/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 15:31:54.449036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:54.448483 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 15:31:55.009088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:55.009050 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:55.449066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:55.449032 5433 patch_prober.go:29] interesting pod/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 15:31:55.449589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:55.449082 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 15:31:55.603909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:55.603868 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:55.750634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:55.750539 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:55.893122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:55.893090 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:56.376700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:56.376669 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:56.449131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:56.449095 5433 patch_prober.go:29] interesting pod/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 15:31:56.449513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:56.449165 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 15:31:56.584418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:56.584384 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:57.314327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:57.314285 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:57.377090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:57.377050 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:31:57.495100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:57.495063 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:31:57.499992 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:57.496770 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:31:58.058252 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:31:58.058226 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:32:00.798509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:00.798468 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl"] Nov 02 15:32:00.798922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:00.798724 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" containerID="cri-o://4bfea21187a8f34216cb5e3c54009833c57f2993fb67baf4486c1dbadeb6a1cf" gracePeriod=30 Nov 02 15:32:00.826202 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:00.826165 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:01.430213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:01.430177 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:01.653474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:01.653437 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:01.793809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:01.793730 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:03.574694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:03.574651 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:04.076202 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:04.076158 5433 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:32:04.076581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:04.076536 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:32:05.272031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:05.260037 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:32:05.319999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:05.317028 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:05.580569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:05.577651 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:05.606038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:05.605108 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:32:05.653999 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:32:05.652799 5433 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.178s" Nov 02 15:32:05.726003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:05.717376 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:32:05.771177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:05.740053 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:32:05.771177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:05.750380 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:32:05.892209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:05.875281 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 15:32:06.350008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.344004 5433 generic.go:356] "Generic (PLEG): container finished" podID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerID="4bfea21187a8f34216cb5e3c54009833c57f2993fb67baf4486c1dbadeb6a1cf" exitCode=0 Nov 02 15:32:06.350008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.344295 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" event={"ID":"668ed6ff-847a-4991-847f-42f45b7bd7d8","Type":"ContainerDied","Data":"4bfea21187a8f34216cb5e3c54009833c57f2993fb67baf4486c1dbadeb6a1cf"} Nov 02 15:32:06.522408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.522374 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:32:06.636963 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.636933 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-token\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-token\") pod \"668ed6ff-847a-4991-847f-42f45b7bd7d8\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " Nov 02 15:32:06.637156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637134 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-certs\") pod \"668ed6ff-847a-4991-847f-42f45b7bd7d8\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " Nov 02 15:32:06.637201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637181 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-podinfo\") pod \"668ed6ff-847a-4991-847f-42f45b7bd7d8\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " Nov 02 15:32:06.637240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637220 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-kube-api-access-n2bwv\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-kube-api-access-n2bwv\") pod \"668ed6ff-847a-4991-847f-42f45b7bd7d8\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " Nov 02 15:32:06.637329 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637313 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-envoy\") pod \"668ed6ff-847a-4991-847f-42f45b7bd7d8\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " Nov 02 15:32:06.637414 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637384 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-socket\") pod \"668ed6ff-847a-4991-847f-42f45b7bd7d8\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " Nov 02 15:32:06.637474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637436 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-certs" pod "668ed6ff-847a-4991-847f-42f45b7bd7d8" (UID: "668ed6ff-847a-4991-847f-42f45b7bd7d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.637474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637447 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-credential-socket\") pod \"668ed6ff-847a-4991-847f-42f45b7bd7d8\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " Nov 02 15:32:06.637583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637531 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-data\") pod \"668ed6ff-847a-4991-847f-42f45b7bd7d8\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " Nov 02 15:32:06.637658 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637635 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668ed6ff-847a-4991-847f-42f45b7bd7d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668ed6ff-847a-4991-847f-42f45b7bd7d8-istiod-ca-cert\") pod \"668ed6ff-847a-4991-847f-42f45b7bd7d8\" (UID: \"668ed6ff-847a-4991-847f-42f45b7bd7d8\") " Nov 02 15:32:06.637716 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637659 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-credential-socket" pod "668ed6ff-847a-4991-847f-42f45b7bd7d8" (UID: "668ed6ff-847a-4991-847f-42f45b7bd7d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.637716 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637675 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-socket" pod "668ed6ff-847a-4991-847f-42f45b7bd7d8" (UID: "668ed6ff-847a-4991-847f-42f45b7bd7d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.637827 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.637765 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-data" pod "668ed6ff-847a-4991-847f-42f45b7bd7d8" (UID: "668ed6ff-847a-4991-847f-42f45b7bd7d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.638132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.638104 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668ed6ff-847a-4991-847f-42f45b7bd7d8-istiod-ca-cert" pod "668ed6ff-847a-4991-847f-42f45b7bd7d8" (UID: "668ed6ff-847a-4991-847f-42f45b7bd7d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:06.638438 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.638412 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.638515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.638443 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.638515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.638459 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.638515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.638475 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.638515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.638491 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668ed6ff-847a-4991-847f-42f45b7bd7d8-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.639740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.639709 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-podinfo" pod "668ed6ff-847a-4991-847f-42f45b7bd7d8" (UID: "668ed6ff-847a-4991-847f-42f45b7bd7d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:32:06.639740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.639718 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-kube-api-access-n2bwv" pod "668ed6ff-847a-4991-847f-42f45b7bd7d8" (UID: "668ed6ff-847a-4991-847f-42f45b7bd7d8"). InnerVolumeSpecName "kube-api-access-n2bwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:06.639914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.639736 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-envoy" pod "668ed6ff-847a-4991-847f-42f45b7bd7d8" (UID: "668ed6ff-847a-4991-847f-42f45b7bd7d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.639914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.639810 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-token" pod "668ed6ff-847a-4991-847f-42f45b7bd7d8" (UID: "668ed6ff-847a-4991-847f-42f45b7bd7d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:06.740141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.740102 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.740141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.740132 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2bwv\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-kube-api-access-n2bwv\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.740141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.740146 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.740401 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:06.740160 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668ed6ff-847a-4991-847f-42f45b7bd7d8-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:07.088395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:07.088298 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:07.352196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:07.352106 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" event={"ID":"668ed6ff-847a-4991-847f-42f45b7bd7d8","Type":"ContainerDied","Data":"6d289ee10907f4259cb612e1d5e08f35b0eea52043f0fc5d0607531bcfaf7275"} Nov 02 15:32:07.352196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:07.352166 5433 scope.go:122] "RemoveContainer" containerID="4bfea21187a8f34216cb5e3c54009833c57f2993fb67baf4486c1dbadeb6a1cf" Nov 02 15:32:07.352196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:07.352121 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl" Nov 02 15:32:07.555628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:07.555588 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl"] Nov 02 15:32:07.599985 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:07.599936 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-x6qpl"] Nov 02 15:32:08.458372 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:08.458338 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" path="/var/lib/kubelet/pods/668ed6ff-847a-4991-847f-42f45b7bd7d8/volumes" Nov 02 15:32:10.955374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:10.955342 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz"] Nov 02 15:32:11.012243 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:32:11.012206 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:32:11.236945 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:11.236790 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:11.437303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:11.437265 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:12.689279 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:12.689243 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:12.917082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:12.917046 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:13.099076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:13.098376 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:15.956582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:15.956539 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" podUID="c1b66d9a-4ea2-4d8f-aeee-e7336428164e" containerName="payload-pre-processing" containerID="cri-o://8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f" gracePeriod=25 Nov 02 15:32:17.963310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:17.963274 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:21.685722 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:21.685686 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" podUID="c1b66d9a-4ea2-4d8f-aeee-e7336428164e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:32:23.080567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:23.080529 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:27.703947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:27.703901 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:28.567958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:28.567924 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:31.729141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:31.729090 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" podUID="c1b66d9a-4ea2-4d8f-aeee-e7336428164e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:32:32.549222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:32.549124 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:32:32.549943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:32.549438 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:32:32.757140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:32.757092 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:32:34.342132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:34.342089 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:36.249094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.249066 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:32:36.352324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.352291 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-kube-api-access-fwnxb\" (UniqueName: \"kubernetes.io/projected/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-kube-api-access-fwnxb\") pod \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\" (UID: \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\") " Nov 02 15:32:36.352502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.352463 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-plugins-config-volume\") pod \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\" (UID: \"c1b66d9a-4ea2-4d8f-aeee-e7336428164e\") " Nov 02 15:32:36.352859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.352822 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-plugins-config-volume" pod "c1b66d9a-4ea2-4d8f-aeee-e7336428164e" (UID: "c1b66d9a-4ea2-4d8f-aeee-e7336428164e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:36.354530 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.354499 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-kube-api-access-fwnxb" pod "c1b66d9a-4ea2-4d8f-aeee-e7336428164e" (UID: "c1b66d9a-4ea2-4d8f-aeee-e7336428164e"). InnerVolumeSpecName "kube-api-access-fwnxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:36.453835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.453789 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:36.453835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.453820 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwnxb\" (UniqueName: \"kubernetes.io/projected/c1b66d9a-4ea2-4d8f-aeee-e7336428164e-kube-api-access-fwnxb\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:36.689681 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.689649 5433 generic.go:356] "Generic (PLEG): container finished" podID="c1b66d9a-4ea2-4d8f-aeee-e7336428164e" containerID="8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f" exitCode=0 Nov 02 15:32:36.689882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.689694 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" Nov 02 15:32:36.689882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.689739 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" event={"ID":"c1b66d9a-4ea2-4d8f-aeee-e7336428164e","Type":"ContainerDied","Data":"8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f"} Nov 02 15:32:36.689882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.689793 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz" event={"ID":"c1b66d9a-4ea2-4d8f-aeee-e7336428164e","Type":"ContainerDied","Data":"d192901897869e0fffe622fac91306f72c88d3cb33b95e7148a3e413b8eeefe2"} Nov 02 15:32:36.689882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.689814 5433 scope.go:122] "RemoveContainer" containerID="8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f" Nov 02 15:32:36.700445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.700426 5433 scope.go:122] "RemoveContainer" containerID="8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f" Nov 02 15:32:36.700737 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:32:36.700718 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f\": container with ID starting with 8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f not found: ID does not exist" containerID="8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f" Nov 02 15:32:36.700780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.700746 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f"} err="failed to get container status \"8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f\": rpc error: code = NotFound desc = could not find container \"8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f\": container with ID starting with 8c5ca1a8183549af9333492ab17426d11679125cce49cbc727f446c9a421482f not found: ID does not exist" Nov 02 15:32:36.845387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.845352 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz"] Nov 02 15:32:36.891441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:36.891399 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-qxdjz"] Nov 02 15:32:38.458048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:38.458013 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b66d9a-4ea2-4d8f-aeee-e7336428164e" path="/var/lib/kubelet/pods/c1b66d9a-4ea2-4d8f-aeee-e7336428164e/volumes" Nov 02 15:32:39.953935 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:39.953872 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:41.013187 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:32:41.013147 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:32:42.369487 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:42.369450 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm"] Nov 02 15:32:42.472129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:42.472096 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb"] Nov 02 15:32:42.716382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:42.716336 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:42.869771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:42.869733 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:45.438529 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:45.438496 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:47.370133 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.370070 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" podUID="ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132" containerName="payload-processing" containerID="cri-o://edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66" gracePeriod=25 Nov 02 15:32:47.472761 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.472718 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" podUID="80b08f02-945f-4a6c-a571-0ad175b5d2c6" containerName="payload-pre-processing" containerID="cri-o://0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5" gracePeriod=25 Nov 02 15:32:47.676723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.676697 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:32:47.738461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.738433 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-plugins-config-volume\") pod \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\" (UID: \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\") " Nov 02 15:32:47.738651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.738497 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-kube-api-access-s4b4g\" (UniqueName: \"kubernetes.io/projected/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-kube-api-access-s4b4g\") pod \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\" (UID: \"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132\") " Nov 02 15:32:47.739231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.739144 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-plugins-config-volume" pod "ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132" (UID: "ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:47.739522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.739496 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:47.741429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.741402 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-kube-api-access-s4b4g" pod "ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132" (UID: "ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132"). InnerVolumeSpecName "kube-api-access-s4b4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:47.762245 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.762224 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:32:47.767904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.767881 5433 generic.go:356] "Generic (PLEG): container finished" podID="80b08f02-945f-4a6c-a571-0ad175b5d2c6" containerID="0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5" exitCode=0 Nov 02 15:32:47.768031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.767917 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" Nov 02 15:32:47.768031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.767964 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" event={"ID":"80b08f02-945f-4a6c-a571-0ad175b5d2c6","Type":"ContainerDied","Data":"0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5"} Nov 02 15:32:47.768151 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.768064 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb" event={"ID":"80b08f02-945f-4a6c-a571-0ad175b5d2c6","Type":"ContainerDied","Data":"48309b78efd70bc6f9fbe65fee14b894c45844cd224a09bc70ee78ad076048fb"} Nov 02 15:32:47.768151 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.768100 5433 scope.go:122] "RemoveContainer" containerID="0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5" Nov 02 15:32:47.769777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.769753 5433 generic.go:356] "Generic (PLEG): container finished" podID="ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132" containerID="edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66" exitCode=0 Nov 02 15:32:47.770045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.769825 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" Nov 02 15:32:47.770045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.769842 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" event={"ID":"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132","Type":"ContainerDied","Data":"edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66"} Nov 02 15:32:47.770202 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.770092 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm" event={"ID":"ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132","Type":"ContainerDied","Data":"86ea239356bb28301875c943fc92ba8b94c5c92855e8112163e38d220e077c1b"} Nov 02 15:32:47.780377 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.780291 5433 scope.go:122] "RemoveContainer" containerID="0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5" Nov 02 15:32:47.782519 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:32:47.782494 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5\": container with ID starting with 0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5 not found: ID does not exist" containerID="0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5" Nov 02 15:32:47.782614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.782525 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5"} err="failed to get container status \"0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5\": rpc error: code = NotFound desc = could not find container \"0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5\": container with ID starting with 0139093d5bb9584f671a2b683ca94018a457725481862744031d3dce4488fda5 not found: ID does not exist" Nov 02 15:32:47.782614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.782543 5433 scope.go:122] "RemoveContainer" containerID="edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66" Nov 02 15:32:47.795178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.795157 5433 scope.go:122] "RemoveContainer" containerID="edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66" Nov 02 15:32:47.795496 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:32:47.795470 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66\": container with ID starting with edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66 not found: ID does not exist" containerID="edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66" Nov 02 15:32:47.795569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.795508 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66"} err="failed to get container status \"edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66\": rpc error: code = NotFound desc = could not find container \"edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66\": container with ID starting with edcfc6a3e553d86cad4a47ae94793e869c73777b9dc73209c182daf5b6f2cb66 not found: ID does not exist" Nov 02 15:32:47.840962 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.840926 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b08f02-945f-4a6c-a571-0ad175b5d2c6-kube-api-access-v7lz7\" (UniqueName: \"kubernetes.io/projected/80b08f02-945f-4a6c-a571-0ad175b5d2c6-kube-api-access-v7lz7\") pod \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\" (UID: \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\") " Nov 02 15:32:47.841183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.841165 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b08f02-945f-4a6c-a571-0ad175b5d2c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b08f02-945f-4a6c-a571-0ad175b5d2c6-plugins-config-volume\") pod \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\" (UID: \"80b08f02-945f-4a6c-a571-0ad175b5d2c6\") " Nov 02 15:32:47.841599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.841558 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b08f02-945f-4a6c-a571-0ad175b5d2c6-plugins-config-volume" pod "80b08f02-945f-4a6c-a571-0ad175b5d2c6" (UID: "80b08f02-945f-4a6c-a571-0ad175b5d2c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:47.841867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.841847 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4b4g\" (UniqueName: \"kubernetes.io/projected/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132-kube-api-access-s4b4g\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:47.841938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.841869 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b08f02-945f-4a6c-a571-0ad175b5d2c6-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:47.843675 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.843645 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b08f02-945f-4a6c-a571-0ad175b5d2c6-kube-api-access-v7lz7" pod "80b08f02-945f-4a6c-a571-0ad175b5d2c6" (UID: "80b08f02-945f-4a6c-a571-0ad175b5d2c6"). InnerVolumeSpecName "kube-api-access-v7lz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:47.892815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.892735 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm"] Nov 02 15:32:47.896741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.896709 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3602ca-58466844f-xg4rm"] Nov 02 15:32:47.942841 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:47.942813 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7lz7\" (UniqueName: \"kubernetes.io/projected/80b08f02-945f-4a6c-a571-0ad175b5d2c6-kube-api-access-v7lz7\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:48.181904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:48.181820 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb"] Nov 02 15:32:48.188644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:48.188611 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3602ca-686b5567d7-vnxbb"] Nov 02 15:32:48.457737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:48.457657 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b08f02-945f-4a6c-a571-0ad175b5d2c6" path="/var/lib/kubelet/pods/80b08f02-945f-4a6c-a571-0ad175b5d2c6/volumes" Nov 02 15:32:48.458184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:48.458164 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132" path="/var/lib/kubelet/pods/ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132/volumes" Nov 02 15:32:49.702869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:49.702826 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:50.789188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:50.789155 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:55.656528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:55.656476 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg"] Nov 02 15:32:55.766042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:55.766002 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88"] Nov 02 15:32:55.866770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:55.866725 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:56.083044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:56.081967 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:32:56.249047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:32:56.247958 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:00.657942 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:00.657873 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" podUID="732a4e6d-1b45-4331-a5f9-ec9fe101079a" containerName="payload-processing" containerID="cri-o://d1b4bd21d743d240a8d94cde2e34b74a6eaa3d4431ff13d7bb07aab017412c28" gracePeriod=25 Nov 02 15:33:00.766926 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:00.766877 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" podUID="62de370c-3787-488d-b6e5-cd6d61d0ef5c" containerName="payload-pre-processing" containerID="cri-o://e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343" gracePeriod=25 Nov 02 15:33:00.866435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:00.866396 5433 generic.go:356] "Generic (PLEG): container finished" podID="732a4e6d-1b45-4331-a5f9-ec9fe101079a" containerID="d1b4bd21d743d240a8d94cde2e34b74a6eaa3d4431ff13d7bb07aab017412c28" exitCode=0 Nov 02 15:33:00.866585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:00.866506 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" event={"ID":"732a4e6d-1b45-4331-a5f9-ec9fe101079a","Type":"ContainerDied","Data":"d1b4bd21d743d240a8d94cde2e34b74a6eaa3d4431ff13d7bb07aab017412c28"} Nov 02 15:33:00.916513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:00.915123 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:00.936022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:00.935959 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:33:01.070503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.066190 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732a4e6d-1b45-4331-a5f9-ec9fe101079a-kube-api-access-55xg5\" (UniqueName: \"kubernetes.io/projected/732a4e6d-1b45-4331-a5f9-ec9fe101079a-kube-api-access-55xg5\") pod \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\" (UID: \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\") " Nov 02 15:33:01.070503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.066395 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732a4e6d-1b45-4331-a5f9-ec9fe101079a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732a4e6d-1b45-4331-a5f9-ec9fe101079a-plugins-config-volume\") pod \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\" (UID: \"732a4e6d-1b45-4331-a5f9-ec9fe101079a\") " Nov 02 15:33:01.070503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.067835 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732a4e6d-1b45-4331-a5f9-ec9fe101079a-plugins-config-volume" pod "732a4e6d-1b45-4331-a5f9-ec9fe101079a" (UID: "732a4e6d-1b45-4331-a5f9-ec9fe101079a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:01.076621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.076593 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732a4e6d-1b45-4331-a5f9-ec9fe101079a-kube-api-access-55xg5" pod "732a4e6d-1b45-4331-a5f9-ec9fe101079a" (UID: "732a4e6d-1b45-4331-a5f9-ec9fe101079a"). InnerVolumeSpecName "kube-api-access-55xg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:01.123789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.123762 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:33:01.168511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.168439 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62de370c-3787-488d-b6e5-cd6d61d0ef5c-kube-api-access-qb2nr\" (UniqueName: \"kubernetes.io/projected/62de370c-3787-488d-b6e5-cd6d61d0ef5c-kube-api-access-qb2nr\") pod \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\" (UID: \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\") " Nov 02 15:33:01.168647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.168550 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62de370c-3787-488d-b6e5-cd6d61d0ef5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de370c-3787-488d-b6e5-cd6d61d0ef5c-plugins-config-volume\") pod \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\" (UID: \"62de370c-3787-488d-b6e5-cd6d61d0ef5c\") " Nov 02 15:33:01.168920 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.168892 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62de370c-3787-488d-b6e5-cd6d61d0ef5c-plugins-config-volume" pod "62de370c-3787-488d-b6e5-cd6d61d0ef5c" (UID: "62de370c-3787-488d-b6e5-cd6d61d0ef5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:01.169258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.169239 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55xg5\" (UniqueName: \"kubernetes.io/projected/732a4e6d-1b45-4331-a5f9-ec9fe101079a-kube-api-access-55xg5\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:01.169307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.169262 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de370c-3787-488d-b6e5-cd6d61d0ef5c-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:01.169307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.169275 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/732a4e6d-1b45-4331-a5f9-ec9fe101079a-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:01.170553 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.170532 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62de370c-3787-488d-b6e5-cd6d61d0ef5c-kube-api-access-qb2nr" pod "62de370c-3787-488d-b6e5-cd6d61d0ef5c" (UID: "62de370c-3787-488d-b6e5-cd6d61d0ef5c"). InnerVolumeSpecName "kube-api-access-qb2nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:01.271366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.271323 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb2nr\" (UniqueName: \"kubernetes.io/projected/62de370c-3787-488d-b6e5-cd6d61d0ef5c-kube-api-access-qb2nr\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:01.874799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.874765 5433 generic.go:356] "Generic (PLEG): container finished" podID="62de370c-3787-488d-b6e5-cd6d61d0ef5c" containerID="e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343" exitCode=0 Nov 02 15:33:01.875292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.874806 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" Nov 02 15:33:01.875292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.874851 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" event={"ID":"62de370c-3787-488d-b6e5-cd6d61d0ef5c","Type":"ContainerDied","Data":"e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343"} Nov 02 15:33:01.875292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.874900 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88" event={"ID":"62de370c-3787-488d-b6e5-cd6d61d0ef5c","Type":"ContainerDied","Data":"a84947230544b87f57b909d4a7acad4e10fb8b8facde6569022e79fbb481f687"} Nov 02 15:33:01.875292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.874921 5433 scope.go:122] "RemoveContainer" containerID="e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343" Nov 02 15:33:01.876441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.876420 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" event={"ID":"732a4e6d-1b45-4331-a5f9-ec9fe101079a","Type":"ContainerDied","Data":"5ab48e42d306e37236010e2f9ea8fb96b28632dd63e8951103a72c4e62693337"} Nov 02 15:33:01.876515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.876452 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg" Nov 02 15:33:01.887430 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.887414 5433 scope.go:122] "RemoveContainer" containerID="e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343" Nov 02 15:33:01.887703 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:33:01.887682 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343\": container with ID starting with e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343 not found: ID does not exist" containerID="e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343" Nov 02 15:33:01.887784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.887714 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343"} err="failed to get container status \"e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343\": rpc error: code = NotFound desc = could not find container \"e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343\": container with ID starting with e45a89621459f2178634238933b1863811fbaf9b93ce5c7e7606e3a4d0c7a343 not found: ID does not exist" Nov 02 15:33:01.887784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:01.887739 5433 scope.go:122] "RemoveContainer" containerID="d1b4bd21d743d240a8d94cde2e34b74a6eaa3d4431ff13d7bb07aab017412c28" Nov 02 15:33:02.030218 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:02.030141 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88"] Nov 02 15:33:02.076187 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:02.076153 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0de358-768b6b9d79-dvr88"] Nov 02 15:33:02.179342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:02.179300 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg"] Nov 02 15:33:02.183650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:02.183614 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0de358-6796674449-fvtqg"] Nov 02 15:33:02.458061 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:02.458021 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62de370c-3787-488d-b6e5-cd6d61d0ef5c" path="/var/lib/kubelet/pods/62de370c-3787-488d-b6e5-cd6d61d0ef5c/volumes" Nov 02 15:33:02.458547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:02.458526 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732a4e6d-1b45-4331-a5f9-ec9fe101079a" path="/var/lib/kubelet/pods/732a4e6d-1b45-4331-a5f9-ec9fe101079a/volumes" Nov 02 15:33:05.806753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:05.806681 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:05.981222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:05.981178 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:06.138555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:06.138502 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:07.411544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:07.411503 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:07.557266 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:07.557233 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:11.122998 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:33:11.120157 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:33:11.127000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:11.125963 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:16.276658 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:16.276618 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:19.684989 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:19.684947 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:19.829887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:19.829849 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:21.258386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:21.258356 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:26.248845 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:26.248808 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:28.043605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:28.043571 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:31.208845 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:31.208805 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:31.379045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:31.379004 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:31.520950 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:31.520843 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:36.686410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:36.686370 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:38.047829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:38.047792 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:41.012018 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:33:41.011960 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:33:41.710703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:41.710664 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:44.287296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:44.287239 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:46.846701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:46.846660 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:51.248626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.248587 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb"] Nov 02 15:33:51.249565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.249526 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" podUID="95e95535-82c8-4251-80d3-da54d4f1c6d2" containerName="maas-api" containerID="cri-o://7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2" gracePeriod=30 Nov 02 15:33:51.450075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.449924 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:51.651810 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.651783 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:33:51.730621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.730586 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e95535-82c8-4251-80d3-da54d4f1c6d2-kube-api-access-259hl\" (UniqueName: \"kubernetes.io/projected/95e95535-82c8-4251-80d3-da54d4f1c6d2-kube-api-access-259hl\") pod \"95e95535-82c8-4251-80d3-da54d4f1c6d2\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " Nov 02 15:33:51.731743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.731712 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls\") pod \"95e95535-82c8-4251-80d3-da54d4f1c6d2\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " Nov 02 15:33:51.731956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.731938 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-metrics-certs\") pod \"95e95535-82c8-4251-80d3-da54d4f1c6d2\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " Nov 02 15:33:51.732890 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.732868 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e95535-82c8-4251-80d3-da54d4f1c6d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e95535-82c8-4251-80d3-da54d4f1c6d2-openshift-service-ca\") pod \"95e95535-82c8-4251-80d3-da54d4f1c6d2\" (UID: \"95e95535-82c8-4251-80d3-da54d4f1c6d2\") " Nov 02 15:33:51.738556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.738529 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e95535-82c8-4251-80d3-da54d4f1c6d2-openshift-service-ca" pod "95e95535-82c8-4251-80d3-da54d4f1c6d2" (UID: "95e95535-82c8-4251-80d3-da54d4f1c6d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:51.749004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.747561 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-metrics-certs" pod "95e95535-82c8-4251-80d3-da54d4f1c6d2" (UID: "95e95535-82c8-4251-80d3-da54d4f1c6d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:33:51.776450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.765858 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e95535-82c8-4251-80d3-da54d4f1c6d2-kube-api-access-259hl" pod "95e95535-82c8-4251-80d3-da54d4f1c6d2" (UID: "95e95535-82c8-4251-80d3-da54d4f1c6d2"). InnerVolumeSpecName "kube-api-access-259hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:51.776450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.766488 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls" pod "95e95535-82c8-4251-80d3-da54d4f1c6d2" (UID: "95e95535-82c8-4251-80d3-da54d4f1c6d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:33:51.835463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.835380 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-259hl\" (UniqueName: \"kubernetes.io/projected/95e95535-82c8-4251-80d3-da54d4f1c6d2-kube-api-access-259hl\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:51.835463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.835414 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:51.835463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.835423 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95e95535-82c8-4251-80d3-da54d4f1c6d2-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:51.835463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:51.835431 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e95535-82c8-4251-80d3-da54d4f1c6d2-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:52.293330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.293297 5433 generic.go:356] "Generic (PLEG): container finished" podID="95e95535-82c8-4251-80d3-da54d4f1c6d2" containerID="7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2" exitCode=0 Nov 02 15:33:52.293766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.293338 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" Nov 02 15:33:52.293766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.293394 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" event={"ID":"95e95535-82c8-4251-80d3-da54d4f1c6d2","Type":"ContainerDied","Data":"7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2"} Nov 02 15:33:52.293766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.293440 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" event={"ID":"95e95535-82c8-4251-80d3-da54d4f1c6d2","Type":"ContainerDied","Data":"32740835bd34c08bb2b35cca7f613d11d63ad87291036ff31d440ad8de1cbfd2"} Nov 02 15:33:52.293766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.293464 5433 scope.go:122] "RemoveContainer" containerID="7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2" Nov 02 15:33:52.304327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.304310 5433 scope.go:122] "RemoveContainer" containerID="7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2" Nov 02 15:33:52.304603 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:33:52.304580 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2\": container with ID starting with 7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2 not found: ID does not exist" containerID="7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2" Nov 02 15:33:52.304680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.304613 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2"} err="failed to get container status \"7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2\": rpc error: code = NotFound desc = could not find container \"7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2\": container with ID starting with 7ccf77891244c9a54296135d5b7966d3b30f7b9be0ca764c9207804e026436a2 not found: ID does not exist" Nov 02 15:33:52.493535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.493494 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb"] Nov 02 15:33:52.496565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.496541 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb"] Nov 02 15:33:52.754327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.754290 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6"] Nov 02 15:33:52.754506 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.754487 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" podUID="b66bc869-5f79-4322-873c-c7aea554c7b7" containerName="maas-api" containerID="cri-o://ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d" gracePeriod=30 Nov 02 15:33:52.890339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:52.890292 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:53.110821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.110795 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:33:53.191860 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.191817 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-metrics-certs\") pod \"b66bc869-5f79-4322-873c-c7aea554c7b7\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " Nov 02 15:33:53.192216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.192192 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls\") pod \"b66bc869-5f79-4322-873c-c7aea554c7b7\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " Nov 02 15:33:53.192391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.192363 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66bc869-5f79-4322-873c-c7aea554c7b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66bc869-5f79-4322-873c-c7aea554c7b7-openshift-service-ca\") pod \"b66bc869-5f79-4322-873c-c7aea554c7b7\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " Nov 02 15:33:53.192536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.192520 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66bc869-5f79-4322-873c-c7aea554c7b7-kube-api-access-75fh2\" (UniqueName: \"kubernetes.io/projected/b66bc869-5f79-4322-873c-c7aea554c7b7-kube-api-access-75fh2\") pod \"b66bc869-5f79-4322-873c-c7aea554c7b7\" (UID: \"b66bc869-5f79-4322-873c-c7aea554c7b7\") " Nov 02 15:33:53.197409 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.197355 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66bc869-5f79-4322-873c-c7aea554c7b7-openshift-service-ca" pod "b66bc869-5f79-4322-873c-c7aea554c7b7" (UID: "b66bc869-5f79-4322-873c-c7aea554c7b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:53.232162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.231514 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:53.259109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.255266 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66bc869-5f79-4322-873c-c7aea554c7b7-kube-api-access-75fh2" pod "b66bc869-5f79-4322-873c-c7aea554c7b7" (UID: "b66bc869-5f79-4322-873c-c7aea554c7b7"). InnerVolumeSpecName "kube-api-access-75fh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:53.268868 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.268810 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls" pod "b66bc869-5f79-4322-873c-c7aea554c7b7" (UID: "b66bc869-5f79-4322-873c-c7aea554c7b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:33:53.269357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.269313 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-metrics-certs" pod "b66bc869-5f79-4322-873c-c7aea554c7b7" (UID: "b66bc869-5f79-4322-873c-c7aea554c7b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:33:53.298033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.294224 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:53.298033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.294258 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66bc869-5f79-4322-873c-c7aea554c7b7-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:53.298033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.294273 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66bc869-5f79-4322-873c-c7aea554c7b7-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:53.298033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.294287 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75fh2\" (UniqueName: \"kubernetes.io/projected/b66bc869-5f79-4322-873c-c7aea554c7b7-kube-api-access-75fh2\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:53.304210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.304178 5433 generic.go:356] "Generic (PLEG): container finished" podID="b66bc869-5f79-4322-873c-c7aea554c7b7" containerID="ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d" exitCode=0 Nov 02 15:33:53.304210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.304247 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" event={"ID":"b66bc869-5f79-4322-873c-c7aea554c7b7","Type":"ContainerDied","Data":"ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d"} Nov 02 15:33:53.304544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.304287 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" event={"ID":"b66bc869-5f79-4322-873c-c7aea554c7b7","Type":"ContainerDied","Data":"18506248f3f5b7f1bd25efb4c3ac0784f8f2ea3be8ecb10be79b78de8b470289"} Nov 02 15:33:53.304544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.304321 5433 scope.go:122] "RemoveContainer" containerID="ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d" Nov 02 15:33:53.304544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.304438 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6" Nov 02 15:33:53.316260 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.316234 5433 scope.go:122] "RemoveContainer" containerID="ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d" Nov 02 15:33:53.316766 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:33:53.316520 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d\": container with ID starting with ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d not found: ID does not exist" containerID="ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d" Nov 02 15:33:53.316766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.316550 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d"} err="failed to get container status \"ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d\": rpc error: code = NotFound desc = could not find container \"ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d\": container with ID starting with ae2db93a028d1c9b96744ae0c988163e36ab3634280953cb9192d801fdf95a6d not found: ID does not exist" Nov 02 15:33:53.499674 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.499635 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6"] Nov 02 15:33:53.505889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.505856 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3602ca-65fb8479cd-n6jw6"] Nov 02 15:33:53.512954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:53.512924 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:54.458288 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:54.458256 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e95535-82c8-4251-80d3-da54d4f1c6d2" path="/var/lib/kubelet/pods/95e95535-82c8-4251-80d3-da54d4f1c6d2/volumes" Nov 02 15:33:54.458782 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:54.458767 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66bc869-5f79-4322-873c-c7aea554c7b7" path="/var/lib/kubelet/pods/b66bc869-5f79-4322-873c-c7aea554c7b7/volumes" Nov 02 15:33:54.576740 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:54.576681 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-z8mnb" podUID="95e95535-82c8-4251-80d3-da54d4f1c6d2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.97:8443/health\": context deadline exceeded" Nov 02 15:33:54.585461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:54.585430 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:54.820747 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:54.820655 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:54.966661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:54.966627 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:55.688223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:55.688189 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:58.274512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:58.274478 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp"] Nov 02 15:33:58.300660 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:58.300626 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:58.372935 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:58.372897 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7"] Nov 02 15:33:58.624434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:58.624399 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:33:58.821255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:33:58.817611 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:00.165497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:00.165459 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:02.087148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:02.087078 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:02.286429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:02.286395 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:02.639999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:02.639815 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:02.802015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:02.795732 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:03.223057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:03.100570 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:03.296221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:03.291725 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" podUID="b577fc20-35be-4771-99e8-177ad3f37d8c" containerName="payload-processing" containerID="cri-o://50461e6be60d0eee87b4c964fa608098e2d3d12498ad388a4bdf73131a58e1c1" gracePeriod=25 Nov 02 15:34:03.373607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:03.373562 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" podUID="cb20b3ef-7cc7-455e-a06b-13c0bed51e24" containerName="payload-pre-processing" containerID="cri-o://7b5df584050e5ba9684d0e2ec3d99c3f7dc30a67691a2255bb7dfb1ae839e123" gracePeriod=25 Nov 02 15:34:03.473378 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:03.464859 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx"] Nov 02 15:34:03.558999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:03.555562 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:34:03.591011 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:03.585794 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" containerID="cri-o://9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47" gracePeriod=30 Nov 02 15:34:03.717713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:03.716156 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:03.756684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:03.743077 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:04.182000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.178663 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:04.564068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.563459 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 15:34:04.622174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.622088 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:04.768272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.768174 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:04.871000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.855806 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 15:34:04.895295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.872576 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:04.920149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.920116 5433 generic.go:356] "Generic (PLEG): container finished" podID="cb20b3ef-7cc7-455e-a06b-13c0bed51e24" containerID="7b5df584050e5ba9684d0e2ec3d99c3f7dc30a67691a2255bb7dfb1ae839e123" exitCode=0 Nov 02 15:34:04.920692 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.920674 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:04.960953 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.960914 5433 generic.go:356] "Generic (PLEG): container finished" podID="b577fc20-35be-4771-99e8-177ad3f37d8c" containerID="50461e6be60d0eee87b4c964fa608098e2d3d12498ad388a4bdf73131a58e1c1" exitCode=0 Nov 02 15:34:04.964848 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.964821 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:34:04.968963 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.966968 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:34:04.969188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.969167 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" event={"ID":"cb20b3ef-7cc7-455e-a06b-13c0bed51e24","Type":"ContainerDied","Data":"7b5df584050e5ba9684d0e2ec3d99c3f7dc30a67691a2255bb7dfb1ae839e123"} Nov 02 15:34:04.970246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.970226 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:34:04.970386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.970369 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" event={"ID":"b577fc20-35be-4771-99e8-177ad3f37d8c","Type":"ContainerDied","Data":"50461e6be60d0eee87b4c964fa608098e2d3d12498ad388a4bdf73131a58e1c1"} Nov 02 15:34:04.971160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.971144 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:34:04.971349 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.971338 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:34:04.972548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:04.972535 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:34:05.225943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.225906 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:05.328665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.328584 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:34:05.359117 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.358933 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc"] Nov 02 15:34:05.360037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360011 5433 cpu_manager.go:414] "containerMap: removing container" podUID="732a4e6d-1b45-4331-a5f9-ec9fe101079a" containerName="payload-processing" Nov 02 15:34:05.360037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360037 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732a4e6d-1b45-4331-a5f9-ec9fe101079a" containerName="payload-processing" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360057 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c1b66d9a-4ea2-4d8f-aeee-e7336428164e" containerName="payload-pre-processing" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360065 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b66d9a-4ea2-4d8f-aeee-e7336428164e" containerName="payload-pre-processing" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360085 5433 cpu_manager.go:414] "containerMap: removing container" podUID="95e95535-82c8-4251-80d3-da54d4f1c6d2" containerName="maas-api" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360092 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e95535-82c8-4251-80d3-da54d4f1c6d2" containerName="maas-api" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360110 5433 cpu_manager.go:414] "containerMap: removing container" podUID="62de370c-3787-488d-b6e5-cd6d61d0ef5c" containerName="payload-pre-processing" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360118 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62de370c-3787-488d-b6e5-cd6d61d0ef5c" containerName="payload-pre-processing" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360132 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b66bc869-5f79-4322-873c-c7aea554c7b7" containerName="maas-api" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360141 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66bc869-5f79-4322-873c-c7aea554c7b7" containerName="maas-api" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360154 5433 cpu_manager.go:414] "containerMap: removing container" podUID="80b08f02-945f-4a6c-a571-0ad175b5d2c6" containerName="payload-pre-processing" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360162 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b08f02-945f-4a6c-a571-0ad175b5d2c6" containerName="payload-pre-processing" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360185 5433 cpu_manager.go:414] "containerMap: removing container" podUID="ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132" containerName="payload-processing" Nov 02 15:34:05.360209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360193 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132" containerName="payload-processing" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360263 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b577fc20-35be-4771-99e8-177ad3f37d8c" containerName="payload-processing" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360274 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b577fc20-35be-4771-99e8-177ad3f37d8c" containerName="payload-processing" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360327 5433 cpu_manager.go:414] "containerMap: removing container" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360335 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360500 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b577fc20-35be-4771-99e8-177ad3f37d8c" containerName="payload-processing" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360524 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="668ed6ff-847a-4991-847f-42f45b7bd7d8" containerName="istio-proxy" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360535 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="732a4e6d-1b45-4331-a5f9-ec9fe101079a" containerName="payload-processing" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360546 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b66d9a-4ea2-4d8f-aeee-e7336428164e" containerName="payload-pre-processing" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360556 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e95535-82c8-4251-80d3-da54d4f1c6d2" containerName="maas-api" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360566 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="62de370c-3787-488d-b6e5-cd6d61d0ef5c" containerName="payload-pre-processing" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360577 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66bc869-5f79-4322-873c-c7aea554c7b7" containerName="maas-api" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360587 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b08f02-945f-4a6c-a571-0ad175b5d2c6" containerName="payload-pre-processing" Nov 02 15:34:05.360774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.360599 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcf7d60-a0ad-4ad9-9a3c-fdc22ff52132" containerName="payload-processing" Nov 02 15:34:05.364659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.363941 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:05.374421 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.374391 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:05.387247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.387224 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:34:05.399370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.399272 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc"] Nov 02 15:34:05.439427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.439400 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-plugins-config-volume\") pod \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\" (UID: \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\") " Nov 02 15:34:05.439697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.439677 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-kube-api-access-7v5p2\" (UniqueName: \"kubernetes.io/projected/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-kube-api-access-7v5p2\") pod \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\" (UID: \"cb20b3ef-7cc7-455e-a06b-13c0bed51e24\") " Nov 02 15:34:05.440098 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.440075 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b577fc20-35be-4771-99e8-177ad3f37d8c-kube-api-access-zcns6\" (UniqueName: \"kubernetes.io/projected/b577fc20-35be-4771-99e8-177ad3f37d8c-kube-api-access-zcns6\") pod \"b577fc20-35be-4771-99e8-177ad3f37d8c\" (UID: \"b577fc20-35be-4771-99e8-177ad3f37d8c\") " Nov 02 15:34:05.440278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.440256 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b577fc20-35be-4771-99e8-177ad3f37d8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b577fc20-35be-4771-99e8-177ad3f37d8c-plugins-config-volume\") pod \"b577fc20-35be-4771-99e8-177ad3f37d8c\" (UID: \"b577fc20-35be-4771-99e8-177ad3f37d8c\") " Nov 02 15:34:05.440883 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.440855 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba9a13a-5bd4-4d29-aa48-b10162823acf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc\" (UID: \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:05.442139 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.442119 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/7ba9a13a-5bd4-4d29-aa48-b10162823acf-kube-api-access-x9mq6\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc\" (UID: \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:05.442355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.441144 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-plugins-config-volume" pod "cb20b3ef-7cc7-455e-a06b-13c0bed51e24" (UID: "cb20b3ef-7cc7-455e-a06b-13c0bed51e24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:05.442460 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.441737 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b577fc20-35be-4771-99e8-177ad3f37d8c-plugins-config-volume" pod "b577fc20-35be-4771-99e8-177ad3f37d8c" (UID: "b577fc20-35be-4771-99e8-177ad3f37d8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:05.471119 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.470820 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5aeac88e\"" Nov 02 15:34:05.475124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.474688 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b577fc20-35be-4771-99e8-177ad3f37d8c-kube-api-access-zcns6" pod "b577fc20-35be-4771-99e8-177ad3f37d8c" (UID: "b577fc20-35be-4771-99e8-177ad3f37d8c"). InnerVolumeSpecName "kube-api-access-zcns6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:05.475597 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.475557 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-kube-api-access-7v5p2" pod "cb20b3ef-7cc7-455e-a06b-13c0bed51e24" (UID: "cb20b3ef-7cc7-455e-a06b-13c0bed51e24"). InnerVolumeSpecName "kube-api-access-7v5p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:05.520599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.491409 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t"] Nov 02 15:34:05.520599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.492396 5433 cpu_manager.go:414] "containerMap: removing container" podUID="cb20b3ef-7cc7-455e-a06b-13c0bed51e24" containerName="payload-pre-processing" Nov 02 15:34:05.520599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.492412 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb20b3ef-7cc7-455e-a06b-13c0bed51e24" containerName="payload-pre-processing" Nov 02 15:34:05.520599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.492620 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb20b3ef-7cc7-455e-a06b-13c0bed51e24" containerName="payload-pre-processing" Nov 02 15:34:05.541154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.541126 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t"] Nov 02 15:34:05.541508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.541436 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:05.543942 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.543917 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba9a13a-5bd4-4d29-aa48-b10162823acf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc\" (UID: \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:05.544917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.544897 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/7ba9a13a-5bd4-4d29-aa48-b10162823acf-kube-api-access-x9mq6\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc\" (UID: \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:05.545384 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.544832 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba9a13a-5bd4-4d29-aa48-b10162823acf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc\" (UID: \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:05.545785 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.545765 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v5p2\" (UniqueName: \"kubernetes.io/projected/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-kube-api-access-7v5p2\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:05.545904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.545891 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcns6\" (UniqueName: \"kubernetes.io/projected/b577fc20-35be-4771-99e8-177ad3f37d8c-kube-api-access-zcns6\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:05.546018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.546005 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b577fc20-35be-4771-99e8-177ad3f37d8c-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:05.546110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.546099 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb20b3ef-7cc7-455e-a06b-13c0bed51e24-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:05.596277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.595575 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5aeac88e-dockercfg-6nzmc\"" Nov 02 15:34:05.597665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.596861 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:05.605852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.605818 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/7ba9a13a-5bd4-4d29-aa48-b10162823acf-kube-api-access-x9mq6\") pod \"payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc\" (UID: \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:05.647562 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.647521 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82934be9-75b5-4023-8013-2ae3c5456011-plugins-config-volume\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t\" (UID: \"82934be9-75b5-4023-8013-2ae3c5456011\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:05.647887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.647860 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gkhb\" (UniqueName: \"kubernetes.io/projected/82934be9-75b5-4023-8013-2ae3c5456011-kube-api-access-9gkhb\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t\" (UID: \"82934be9-75b5-4023-8013-2ae3c5456011\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:05.738418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.738386 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:05.749145 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.749109 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gkhb\" (UniqueName: \"kubernetes.io/projected/82934be9-75b5-4023-8013-2ae3c5456011-kube-api-access-9gkhb\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t\" (UID: \"82934be9-75b5-4023-8013-2ae3c5456011\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:05.749288 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.749252 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82934be9-75b5-4023-8013-2ae3c5456011-plugins-config-volume\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t\" (UID: \"82934be9-75b5-4023-8013-2ae3c5456011\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:05.750342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.750314 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82934be9-75b5-4023-8013-2ae3c5456011-plugins-config-volume\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t\" (UID: \"82934be9-75b5-4023-8013-2ae3c5456011\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:05.842044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.835793 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gkhb\" (UniqueName: \"kubernetes.io/projected/82934be9-75b5-4023-8013-2ae3c5456011-kube-api-access-9gkhb\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t\" (UID: \"82934be9-75b5-4023-8013-2ae3c5456011\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:05.913994 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:05.909390 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:06.151484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.151449 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" event={"ID":"cb20b3ef-7cc7-455e-a06b-13c0bed51e24","Type":"ContainerDied","Data":"45c1e9af2d0db98cd935c6989d9cd3dd493935b0371dd5cbf4ea667d35f4af97"} Nov 02 15:34:06.151739 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.151721 5433 scope.go:122] "RemoveContainer" containerID="7b5df584050e5ba9684d0e2ec3d99c3f7dc30a67691a2255bb7dfb1ae839e123" Nov 02 15:34:06.152467 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.152437 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7" Nov 02 15:34:06.162625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.162592 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" event={"ID":"b577fc20-35be-4771-99e8-177ad3f37d8c","Type":"ContainerDied","Data":"10ed1d8e9cb1e900ecac763086ca9743b88845311bec4588bdd6187cc09c56d8"} Nov 02 15:34:06.162736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.162720 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp" Nov 02 15:34:06.214530 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.204437 5433 scope.go:122] "RemoveContainer" containerID="50461e6be60d0eee87b4c964fa608098e2d3d12498ad388a4bdf73131a58e1c1" Nov 02 15:34:06.231659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.231633 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2"] Nov 02 15:34:06.266782 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.266749 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.286017 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.283857 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2"] Nov 02 15:34:06.301240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.299422 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc"] Nov 02 15:34:06.302728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.302690 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:06.303766 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:34:06.303740 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba9a13a_5bd4_4d29_aa48_b10162823acf.slice/crio-7d0b9b5ac55c77b747e21aa5fd5a28ace8f9416f736d991cb01ed9e7c30a6d03 WatchSource:0}: Error finding container 7d0b9b5ac55c77b747e21aa5fd5a28ace8f9416f736d991cb01ed9e7c30a6d03: Status 404 returned error can't find the container with id 7d0b9b5ac55c77b747e21aa5fd5a28ace8f9416f736d991cb01ed9e7c30a6d03 Nov 02 15:34:06.340302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.340243 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-aa5832-openshift-default-dockercfg-99ltt\"" Nov 02 15:34:06.399735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.399707 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-credential-socket\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.399864 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.399750 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-envoy\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.399864 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.399775 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232e57c9-f425-4638-89df-2eadc6a7821b-istio-podinfo\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.399949 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.399920 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232e57c9-f425-4638-89df-2eadc6a7821b-istiod-ca-cert\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.400019 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.399963 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-socket\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.400115 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.400102 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-certs\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.400181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.400168 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-istio-token\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.400268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.400251 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-data\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.400313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.400287 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5c5\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-kube-api-access-8h5c5\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.489928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.489884 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:06.503013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.502837 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-certs\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.503013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.502965 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-istio-token\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.503298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.503054 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-data\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.503298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.503091 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5c5\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-kube-api-access-8h5c5\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.503298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.503214 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-credential-socket\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.503298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.503247 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-envoy\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.503298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.503280 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232e57c9-f425-4638-89df-2eadc6a7821b-istio-podinfo\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.503569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.503358 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232e57c9-f425-4638-89df-2eadc6a7821b-istiod-ca-cert\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.503569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.503409 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-socket\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.508004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.503922 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-socket\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.508004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.504378 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-certs\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.508004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.504897 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-data\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.521468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.520427 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-credential-socket\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.521468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.521397 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232e57c9-f425-4638-89df-2eadc6a7821b-istiod-ca-cert\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.530071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.529199 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-envoy\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.545004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.538717 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232e57c9-f425-4638-89df-2eadc6a7821b-istio-podinfo\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.559397 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.556797 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp"] Nov 02 15:34:06.559578 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.559528 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-941182-79cfd6b659-ls6xp"] Nov 02 15:34:06.562383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.562357 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-istio-token\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.564159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.564136 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t"] Nov 02 15:34:06.577514 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.577491 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5c5\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-kube-api-access-8h5c5\") pod \"e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.582516 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.582494 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:06.611940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.611870 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7"] Nov 02 15:34:06.615574 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:06.615539 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-941182-5b49c677f9-7ltd7"] Nov 02 15:34:07.036385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.036352 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2"] Nov 02 15:34:07.038059 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:34:07.038012 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232e57c9_f425_4638_89df_2eadc6a7821b.slice/crio-82fa27539963743ba4673bc3554257dd727405977bbc37e67e71fec1e7fff5a5 WatchSource:0}: Error finding container 82fa27539963743ba4673bc3554257dd727405977bbc37e67e71fec1e7fff5a5: Status 404 returned error can't find the container with id 82fa27539963743ba4673bc3554257dd727405977bbc37e67e71fec1e7fff5a5 Nov 02 15:34:07.041491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.041455 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:07.041576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.041540 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:07.041622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.041579 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:07.173825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.173774 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" event={"ID":"7ba9a13a-5bd4-4d29-aa48-b10162823acf","Type":"ContainerStarted","Data":"d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b"} Nov 02 15:34:07.174032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.173843 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" event={"ID":"7ba9a13a-5bd4-4d29-aa48-b10162823acf","Type":"ContainerStarted","Data":"7d0b9b5ac55c77b747e21aa5fd5a28ace8f9416f736d991cb01ed9e7c30a6d03"} Nov 02 15:34:07.174032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.174003 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:07.177993 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.175906 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" event={"ID":"232e57c9-f425-4638-89df-2eadc6a7821b","Type":"ContainerStarted","Data":"91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb"} Nov 02 15:34:07.177993 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.176007 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" event={"ID":"232e57c9-f425-4638-89df-2eadc6a7821b","Type":"ContainerStarted","Data":"82fa27539963743ba4673bc3554257dd727405977bbc37e67e71fec1e7fff5a5"} Nov 02 15:34:07.179780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.179749 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" event={"ID":"82934be9-75b5-4023-8013-2ae3c5456011","Type":"ContainerStarted","Data":"3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2"} Nov 02 15:34:07.179905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.179792 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" event={"ID":"82934be9-75b5-4023-8013-2ae3c5456011","Type":"ContainerStarted","Data":"e96d76fe3f71916a9dd368d6814cc52bf53592adcbb5c8253dc199ab955f9bcb"} Nov 02 15:34:07.179905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.179868 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:07.292266 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.292149 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" podStartSLOduration=2.2921296509999998 podStartE2EDuration="2.292129651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:07.288932489 +0000 UTC m=+2150.962135667" watchObservedRunningTime="2026-11-02 15:34:07.292129651 +0000 UTC m=+2150.965332848" Nov 02 15:34:07.437922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.437862 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" podStartSLOduration=2.437829281 podStartE2EDuration="2.437829281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:07.397831597 +0000 UTC m=+2151.071034793" watchObservedRunningTime="2026-11-02 15:34:07.437829281 +0000 UTC m=+2151.111032477" Nov 02 15:34:07.554387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.554266 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" podStartSLOduration=1.55424714 podStartE2EDuration="1.55424714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:07.513504168 +0000 UTC m=+2151.186707363" watchObservedRunningTime="2026-11-02 15:34:07.55424714 +0000 UTC m=+2151.227450339" Nov 02 15:34:07.589246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.589210 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:07.967540 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:07.967503 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:08.118411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.118380 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:08.414344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.414310 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:08.495568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.495533 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b577fc20-35be-4771-99e8-177ad3f37d8c" path="/var/lib/kubelet/pods/b577fc20-35be-4771-99e8-177ad3f37d8c/volumes" Nov 02 15:34:08.496123 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.496100 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb20b3ef-7cc7-455e-a06b-13c0bed51e24" path="/var/lib/kubelet/pods/cb20b3ef-7cc7-455e-a06b-13c0bed51e24/volumes" Nov 02 15:34:08.589429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.589398 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:08.883572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.883542 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:34:08.958826 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.958795 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd811f1-973b-4846-acda-1d3e2b2a2042-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd811f1-973b-4846-acda-1d3e2b2a2042-istiod-ca-cert\") pod \"ecd811f1-973b-4846-acda-1d3e2b2a2042\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " Nov 02 15:34:08.958959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.958849 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-token\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-token\") pod \"ecd811f1-973b-4846-acda-1d3e2b2a2042\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " Nov 02 15:34:08.958959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.958874 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-data\") pod \"ecd811f1-973b-4846-acda-1d3e2b2a2042\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " Nov 02 15:34:08.958959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.958892 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-credential-socket\") pod \"ecd811f1-973b-4846-acda-1d3e2b2a2042\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " Nov 02 15:34:08.958959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.958929 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-podinfo\") pod \"ecd811f1-973b-4846-acda-1d3e2b2a2042\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " Nov 02 15:34:08.959162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959038 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-socket\") pod \"ecd811f1-973b-4846-acda-1d3e2b2a2042\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " Nov 02 15:34:08.959162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959124 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-kube-api-access-rn6m7\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-kube-api-access-rn6m7\") pod \"ecd811f1-973b-4846-acda-1d3e2b2a2042\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " Nov 02 15:34:08.959267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959176 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-certs\") pod \"ecd811f1-973b-4846-acda-1d3e2b2a2042\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " Nov 02 15:34:08.959267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959223 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-data" pod "ecd811f1-973b-4846-acda-1d3e2b2a2042" (UID: "ecd811f1-973b-4846-acda-1d3e2b2a2042"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:08.959386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959299 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-credential-socket" pod "ecd811f1-973b-4846-acda-1d3e2b2a2042" (UID: "ecd811f1-973b-4846-acda-1d3e2b2a2042"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:08.959386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959344 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd811f1-973b-4846-acda-1d3e2b2a2042-istiod-ca-cert" pod "ecd811f1-973b-4846-acda-1d3e2b2a2042" (UID: "ecd811f1-973b-4846-acda-1d3e2b2a2042"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:08.959386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959362 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-envoy\") pod \"ecd811f1-973b-4846-acda-1d3e2b2a2042\" (UID: \"ecd811f1-973b-4846-acda-1d3e2b2a2042\") " Nov 02 15:34:08.959546 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959478 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-certs" pod "ecd811f1-973b-4846-acda-1d3e2b2a2042" (UID: "ecd811f1-973b-4846-acda-1d3e2b2a2042"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:08.959617 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959593 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-socket" pod "ecd811f1-973b-4846-acda-1d3e2b2a2042" (UID: "ecd811f1-973b-4846-acda-1d3e2b2a2042"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:08.959966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959946 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:08.959966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.959966 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd811f1-973b-4846-acda-1d3e2b2a2042-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:08.960167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.960002 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:08.960167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.960012 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:08.960167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.960021 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:08.962012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.961871 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-envoy" pod "ecd811f1-973b-4846-acda-1d3e2b2a2042" (UID: "ecd811f1-973b-4846-acda-1d3e2b2a2042"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:08.962012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.961952 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-podinfo" pod "ecd811f1-973b-4846-acda-1d3e2b2a2042" (UID: "ecd811f1-973b-4846-acda-1d3e2b2a2042"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:34:08.962012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.961962 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-kube-api-access-rn6m7" pod "ecd811f1-973b-4846-acda-1d3e2b2a2042" (UID: "ecd811f1-973b-4846-acda-1d3e2b2a2042"). InnerVolumeSpecName "kube-api-access-rn6m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:08.962171 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:08.962154 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-token" pod "ecd811f1-973b-4846-acda-1d3e2b2a2042" (UID: "ecd811f1-973b-4846-acda-1d3e2b2a2042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:09.060734 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.060652 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:09.061156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.060965 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:09.061156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.061000 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:09.061156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.061010 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn6m7\" (UniqueName: \"kubernetes.io/projected/ecd811f1-973b-4846-acda-1d3e2b2a2042-kube-api-access-rn6m7\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:09.061156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.061019 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd811f1-973b-4846-acda-1d3e2b2a2042-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:09.201137 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.201097 5433 generic.go:356] "Generic (PLEG): container finished" podID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerID="9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47" exitCode=0 Nov 02 15:34:09.201319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.201155 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" Nov 02 15:34:09.201319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.201180 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" event={"ID":"ecd811f1-973b-4846-acda-1d3e2b2a2042","Type":"ContainerDied","Data":"9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47"} Nov 02 15:34:09.201319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.201234 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx" event={"ID":"ecd811f1-973b-4846-acda-1d3e2b2a2042","Type":"ContainerDied","Data":"a0a9c07e78ee0b96456b0ad15ce5a8d2a3426d0460944d5014051dfb6875b7bc"} Nov 02 15:34:09.201319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.201258 5433 scope.go:122] "RemoveContainer" containerID="9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47" Nov 02 15:34:09.201739 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.201721 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:09.202824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.202805 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:34:09.212417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.212399 5433 scope.go:122] "RemoveContainer" containerID="b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6" Nov 02 15:34:09.222233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.222216 5433 scope.go:122] "RemoveContainer" containerID="9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47" Nov 02 15:34:09.222547 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:34:09.222528 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47\": container with ID starting with 9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47 not found: ID does not exist" containerID="9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47" Nov 02 15:34:09.222632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.222564 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47"} err="failed to get container status \"9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47\": rpc error: code = NotFound desc = could not find container \"9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47\": container with ID starting with 9b666f8c68f7d88ef0c379cffd71836d5e96167711966ab71a83e80be3f73a47 not found: ID does not exist" Nov 02 15:34:09.222632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.222583 5433 scope.go:122] "RemoveContainer" containerID="b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6" Nov 02 15:34:09.222872 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:34:09.222852 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6\": container with ID starting with b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6 not found: ID does not exist" containerID="b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6" Nov 02 15:34:09.222918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.222879 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6"} err="failed to get container status \"b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6\": rpc error: code = NotFound desc = could not find container \"b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6\": container with ID starting with b886f1b02758cea3805577204e292e8baf350968a62636dcaf331f29186831c6 not found: ID does not exist" Nov 02 15:34:09.356474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.356394 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx"] Nov 02 15:34:09.402251 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:09.402215 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3602ca-openshift-default-5b9759546c-8x8lx"] Nov 02 15:34:10.373440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:10.373374 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:10.457793 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:10.457762 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" path="/var/lib/kubelet/pods/ecd811f1-973b-4846-acda-1d3e2b2a2042/volumes" Nov 02 15:34:10.571389 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:10.571346 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:10.718589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:10.718552 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:11.014769 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:34:11.014675 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:34:13.359027 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.358969 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb"] Nov 02 15:34:13.359765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.359748 5433 cpu_manager.go:414] "containerMap: removing container" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" Nov 02 15:34:13.359812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.359766 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" Nov 02 15:34:13.359905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.359896 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" Nov 02 15:34:13.363792 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.363771 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.401228 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.401199 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:13.405445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.405418 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8"] Nov 02 15:34:13.406141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.406126 5433 cpu_manager.go:414] "containerMap: removing container" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" Nov 02 15:34:13.406201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.406143 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" Nov 02 15:34:13.406313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.406304 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd811f1-973b-4846-acda-1d3e2b2a2042" containerName="istio-proxy" Nov 02 15:34:13.409307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.409290 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.441002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.439867 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-5afebc-openshift-default-dockercfg-57h7j\"" Nov 02 15:34:13.541782 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.541739 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9tpq\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-kube-api-access-k9tpq\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.541940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.541815 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.541940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.541851 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a021967-af9b-4b44-b323-04e82c9e28da-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.541940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.541926 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.542099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542024 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a021967-af9b-4b44-b323-04e82c9e28da-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.542131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542099 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.542168 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542141 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.542209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542177 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.542268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542253 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.542303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542286 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.542365 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542351 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.542419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542405 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.542475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542451 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.542543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542508 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kt2n\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-kube-api-access-7kt2n\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.542665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542647 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.542737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542694 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.542796 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542755 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.542903 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.542881 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.543950 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.543923 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb"] Nov 02 15:34:13.575228 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.575194 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8"] Nov 02 15:34:13.583842 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.583813 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:13.644482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644446 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.644678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644525 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9tpq\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-kube-api-access-k9tpq\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.644678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644553 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.644678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644573 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a021967-af9b-4b44-b323-04e82c9e28da-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.644678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644632 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.644678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644664 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a021967-af9b-4b44-b323-04e82c9e28da-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.644968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644720 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.644968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644754 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.644968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644783 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.644968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644845 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.644968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644871 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.644968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.644950 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.645306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645021 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.645306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645070 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.645306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645083 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.645306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645183 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kt2n\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-kube-api-access-7kt2n\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.645306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645238 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.645306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645302 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.645632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645315 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.645632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645384 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.645632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645443 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.645632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645497 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.645632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645526 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a021967-af9b-4b44-b323-04e82c9e28da-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.645892 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645846 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.645965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.645928 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.646427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.646398 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.646680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.646639 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.646818 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.646796 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.647630 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.647598 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.647741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.647719 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a021967-af9b-4b44-b323-04e82c9e28da-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.647791 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.647751 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.648200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.648184 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.780427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.768707 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kt2n\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-kube-api-access-7kt2n\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.780427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.770877 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:13.810502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.801437 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.810502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.809989 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:13.819047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.816772 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9tpq\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-kube-api-access-k9tpq\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.981285 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.979463 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:13.993621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.993587 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk"] Nov 02 15:34:13.999021 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:13.998969 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:14.025362 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.025293 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:14.050215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.050010 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-aa5832-dockercfg-mcldg\"" Nov 02 15:34:14.057106 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.056295 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk"] Nov 02 15:34:14.112951 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.112694 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b887d84-52a2-415a-99c7-4bb5de296ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk\" (UID: \"4b887d84-52a2-415a-99c7-4bb5de296ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:14.112951 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.112782 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2rhl\" (UniqueName: \"kubernetes.io/projected/4b887d84-52a2-415a-99c7-4bb5de296ba3-kube-api-access-f2rhl\") pod \"payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk\" (UID: \"4b887d84-52a2-415a-99c7-4bb5de296ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:14.117494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.117239 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-aa5832\"" Nov 02 15:34:14.233566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.233461 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2rhl\" (UniqueName: \"kubernetes.io/projected/4b887d84-52a2-415a-99c7-4bb5de296ba3-kube-api-access-f2rhl\") pod \"payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk\" (UID: \"4b887d84-52a2-415a-99c7-4bb5de296ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:14.234110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.234069 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b887d84-52a2-415a-99c7-4bb5de296ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk\" (UID: \"4b887d84-52a2-415a-99c7-4bb5de296ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:14.235266 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.235231 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b887d84-52a2-415a-99c7-4bb5de296ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk\" (UID: \"4b887d84-52a2-415a-99c7-4bb5de296ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:14.342310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.342280 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2rhl\" (UniqueName: \"kubernetes.io/projected/4b887d84-52a2-415a-99c7-4bb5de296ba3-kube-api-access-f2rhl\") pod \"payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk\" (UID: \"4b887d84-52a2-415a-99c7-4bb5de296ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:14.436277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.436251 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb"] Nov 02 15:34:14.439024 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:34:14.437924 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a021967_af9b_4b44_b323_04e82c9e28da.slice/crio-1afa6e32e74bb96d5802022b1c9f5475565a55ddef1adad0096fdf59be633556 WatchSource:0}: Error finding container 1afa6e32e74bb96d5802022b1c9f5475565a55ddef1adad0096fdf59be633556: Status 404 returned error can't find the container with id 1afa6e32e74bb96d5802022b1c9f5475565a55ddef1adad0096fdf59be633556 Nov 02 15:34:14.440839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.440803 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.440921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.440875 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.440921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.440907 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.620481 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.620448 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:14.638323 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.638294 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8"] Nov 02 15:34:14.639093 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:34:14.639066 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5eba7ca_7738_4cb2_a21d_dd1bee386316.slice/crio-37433187b143635b9440d765feb212dfb9b69b2d70c7dd7b7c1e35872f931f7d WatchSource:0}: Error finding container 37433187b143635b9440d765feb212dfb9b69b2d70c7dd7b7c1e35872f931f7d: Status 404 returned error can't find the container with id 37433187b143635b9440d765feb212dfb9b69b2d70c7dd7b7c1e35872f931f7d Nov 02 15:34:14.642559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.642518 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.642655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.642622 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.642712 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:14.642672 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:15.001372 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.001339 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk"] Nov 02 15:34:15.003166 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:34:15.003134 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b887d84_52a2_415a_99c7_4bb5de296ba3.slice/crio-bd6bc06b1d508022e00d2cb7d1e305959d0ef102a333079bcc67ecce15f5df83 WatchSource:0}: Error finding container bd6bc06b1d508022e00d2cb7d1e305959d0ef102a333079bcc67ecce15f5df83: Status 404 returned error can't find the container with id bd6bc06b1d508022e00d2cb7d1e305959d0ef102a333079bcc67ecce15f5df83 Nov 02 15:34:15.380180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.380141 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" event={"ID":"8a021967-af9b-4b44-b323-04e82c9e28da","Type":"ContainerStarted","Data":"19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3"} Nov 02 15:34:15.380343 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.380212 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" event={"ID":"8a021967-af9b-4b44-b323-04e82c9e28da","Type":"ContainerStarted","Data":"1afa6e32e74bb96d5802022b1c9f5475565a55ddef1adad0096fdf59be633556"} Nov 02 15:34:15.381987 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.381950 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" event={"ID":"4b887d84-52a2-415a-99c7-4bb5de296ba3","Type":"ContainerStarted","Data":"f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd"} Nov 02 15:34:15.382118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.382043 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" event={"ID":"4b887d84-52a2-415a-99c7-4bb5de296ba3","Type":"ContainerStarted","Data":"bd6bc06b1d508022e00d2cb7d1e305959d0ef102a333079bcc67ecce15f5df83"} Nov 02 15:34:15.382189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.382166 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:15.383849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.383825 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" event={"ID":"b5eba7ca-7738-4cb2-a21d-dd1bee386316","Type":"ContainerStarted","Data":"1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44"} Nov 02 15:34:15.383935 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.383866 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" event={"ID":"b5eba7ca-7738-4cb2-a21d-dd1bee386316","Type":"ContainerStarted","Data":"37433187b143635b9440d765feb212dfb9b69b2d70c7dd7b7c1e35872f931f7d"} Nov 02 15:34:15.509995 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.509493 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" podStartSLOduration=2.5094706909999998 podStartE2EDuration="2.509470691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:15.50255152 +0000 UTC m=+2159.175754722" watchObservedRunningTime="2026-11-02 15:34:15.509470691 +0000 UTC m=+2159.182673892" Nov 02 15:34:15.773450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.771496 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" podStartSLOduration=2.771476124 podStartE2EDuration="2.771476124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:15.657118026 +0000 UTC m=+2159.330321223" watchObservedRunningTime="2026-11-02 15:34:15.771476124 +0000 UTC m=+2159.444679322" Nov 02 15:34:15.773450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.772437 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" podStartSLOduration=2.772421739 podStartE2EDuration="2.772421739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:15.769407876 +0000 UTC m=+2159.442611077" watchObservedRunningTime="2026-11-02 15:34:15.772421739 +0000 UTC m=+2159.445624934" Nov 02 15:34:15.983998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.982722 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:15.983998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.982892 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:15.983998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:15.982939 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 15:34:16.029428 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:16.029308 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:16.048084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:16.048049 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 15:34:16.049408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:16.049370 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 15:34:16.745997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:16.745860 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:16.981029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:16.980515 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:16.981029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:16.980596 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 15:34:17.014641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:17.014562 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:17.071001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:17.042368 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 15:34:17.071001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:17.042439 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 15:34:17.204013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:17.201562 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:17.980944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:17.980914 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:17.981326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:17.980964 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 15:34:18.054999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:18.039957 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 15:34:18.054999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:18.040037 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 15:34:18.197346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:18.197315 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:34:18.198282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:18.198258 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:34:18.962999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:18.962546 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:18.980786 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:18.980750 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:18.980930 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:18.980797 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 15:34:19.050999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:19.048108 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 15:34:19.050999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:19.048185 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 15:34:19.393388 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:19.393353 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:19.985272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:19.985238 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:19.985551 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:19.985514 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 15:34:20.066637 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:20.064761 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 15:34:20.066637 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:20.064838 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 15:34:20.283784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:20.283693 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:20.560053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:20.543345 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:20.980432 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:20.980391 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:34:20.980621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:20.980453 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 15:34:21.038998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:21.036018 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 15:34:21.038998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:21.036075 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 15:34:21.984557 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:21.984527 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:21.984956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:21.984926 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:21.985691 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:21.985673 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:34:22.031841 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:22.031811 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:22.032123 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:22.032102 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:22.032787 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:22.032766 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:34:24.238296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:24.238256 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:24.868149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:24.868117 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:25.047197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:25.047167 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:25.195782 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:25.195743 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:26.393744 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:26.393669 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:34:28.112578 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:28.112536 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:28.329390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:28.329347 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:28.502286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:28.502248 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:30.022460 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:30.022429 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:35.463564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:35.463525 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:36.450054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:36.450000 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:40.889742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:40.889705 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:41.020026 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:34:41.019951 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:34:44.451892 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:44.451856 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:44.602238 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:44.602205 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:46.284294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:46.284248 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:46.621115 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:46.621030 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w"] Nov 02 15:34:46.723234 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:46.723188 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6"] Nov 02 15:34:46.966720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:46.966689 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:47.133693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:47.133635 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:51.621841 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.621799 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" podUID="e920348e-e97f-4d71-901e-af22e14ff51f" containerName="payload-processing" containerID="cri-o://0d4c7b30d0d0d32e1babfe8d93584c25427161245b283e0304554bcdea2334cc" gracePeriod=25 Nov 02 15:34:51.724033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.723961 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" podUID="c507e533-234e-46ac-b071-5718b372f7b6" containerName="payload-pre-processing" containerID="cri-o://a0291592d19be8b048a1c2f2e9bdd60b51c9ed536a6e3d7887d01f8dae8d7878" gracePeriod=25 Nov 02 15:34:51.752928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.752900 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:51.974089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.974058 5433 generic.go:356] "Generic (PLEG): container finished" podID="e920348e-e97f-4d71-901e-af22e14ff51f" containerID="0d4c7b30d0d0d32e1babfe8d93584c25427161245b283e0304554bcdea2334cc" exitCode=0 Nov 02 15:34:51.974278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.974136 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" event={"ID":"e920348e-e97f-4d71-901e-af22e14ff51f","Type":"ContainerDied","Data":"0d4c7b30d0d0d32e1babfe8d93584c25427161245b283e0304554bcdea2334cc"} Nov 02 15:34:51.974278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.974199 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" event={"ID":"e920348e-e97f-4d71-901e-af22e14ff51f","Type":"ContainerDied","Data":"867d31ef0b28a77423730a16aba4beb272a80e039081a3fa18e6c68e77918bfa"} Nov 02 15:34:51.974278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.974225 5433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="867d31ef0b28a77423730a16aba4beb272a80e039081a3fa18e6c68e77918bfa" Nov 02 15:34:51.975745 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.975723 5433 generic.go:356] "Generic (PLEG): container finished" podID="c507e533-234e-46ac-b071-5718b372f7b6" containerID="a0291592d19be8b048a1c2f2e9bdd60b51c9ed536a6e3d7887d01f8dae8d7878" exitCode=0 Nov 02 15:34:51.976094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.975830 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" event={"ID":"c507e533-234e-46ac-b071-5718b372f7b6","Type":"ContainerDied","Data":"a0291592d19be8b048a1c2f2e9bdd60b51c9ed536a6e3d7887d01f8dae8d7878"} Nov 02 15:34:51.987424 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:51.986180 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:34:52.068737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.068710 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:34:52.087770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.087738 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e920348e-e97f-4d71-901e-af22e14ff51f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e920348e-e97f-4d71-901e-af22e14ff51f-plugins-config-volume\") pod \"e920348e-e97f-4d71-901e-af22e14ff51f\" (UID: \"e920348e-e97f-4d71-901e-af22e14ff51f\") " Nov 02 15:34:52.088027 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.088005 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e920348e-e97f-4d71-901e-af22e14ff51f-kube-api-access-w5l72\" (UniqueName: \"kubernetes.io/projected/e920348e-e97f-4d71-901e-af22e14ff51f-kube-api-access-w5l72\") pod \"e920348e-e97f-4d71-901e-af22e14ff51f\" (UID: \"e920348e-e97f-4d71-901e-af22e14ff51f\") " Nov 02 15:34:52.088147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.088128 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e920348e-e97f-4d71-901e-af22e14ff51f-plugins-config-volume" pod "e920348e-e97f-4d71-901e-af22e14ff51f" (UID: "e920348e-e97f-4d71-901e-af22e14ff51f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:52.088549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.088526 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e920348e-e97f-4d71-901e-af22e14ff51f-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:52.090002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.089967 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e920348e-e97f-4d71-901e-af22e14ff51f-kube-api-access-w5l72" pod "e920348e-e97f-4d71-901e-af22e14ff51f" (UID: "e920348e-e97f-4d71-901e-af22e14ff51f"). InnerVolumeSpecName "kube-api-access-w5l72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:52.189426 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.189343 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c507e533-234e-46ac-b071-5718b372f7b6-kube-api-access-fr7sr\" (UniqueName: \"kubernetes.io/projected/c507e533-234e-46ac-b071-5718b372f7b6-kube-api-access-fr7sr\") pod \"c507e533-234e-46ac-b071-5718b372f7b6\" (UID: \"c507e533-234e-46ac-b071-5718b372f7b6\") " Nov 02 15:34:52.189579 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.189527 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c507e533-234e-46ac-b071-5718b372f7b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507e533-234e-46ac-b071-5718b372f7b6-plugins-config-volume\") pod \"c507e533-234e-46ac-b071-5718b372f7b6\" (UID: \"c507e533-234e-46ac-b071-5718b372f7b6\") " Nov 02 15:34:52.189911 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.189876 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c507e533-234e-46ac-b071-5718b372f7b6-plugins-config-volume" pod "c507e533-234e-46ac-b071-5718b372f7b6" (UID: "c507e533-234e-46ac-b071-5718b372f7b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:52.190054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.190036 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5l72\" (UniqueName: \"kubernetes.io/projected/e920348e-e97f-4d71-901e-af22e14ff51f-kube-api-access-w5l72\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:52.190054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.190051 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c507e533-234e-46ac-b071-5718b372f7b6-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:52.191449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.191429 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c507e533-234e-46ac-b071-5718b372f7b6-kube-api-access-fr7sr" pod "c507e533-234e-46ac-b071-5718b372f7b6" (UID: "c507e533-234e-46ac-b071-5718b372f7b6"). InnerVolumeSpecName "kube-api-access-fr7sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:52.291591 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.291558 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr7sr\" (UniqueName: \"kubernetes.io/projected/c507e533-234e-46ac-b071-5718b372f7b6-kube-api-access-fr7sr\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:52.984927 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.984894 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" Nov 02 15:34:52.985394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.984929 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6" event={"ID":"c507e533-234e-46ac-b071-5718b372f7b6","Type":"ContainerDied","Data":"61b70fbfcd5a65260f51aaea10675023b0dcb9dd371abb8a178edb1c366473f5"} Nov 02 15:34:52.985394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.985028 5433 scope.go:122] "RemoveContainer" containerID="a0291592d19be8b048a1c2f2e9bdd60b51c9ed536a6e3d7887d01f8dae8d7878" Nov 02 15:34:52.985394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:52.985091 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w" Nov 02 15:34:53.188401 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:53.188367 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6"] Nov 02 15:34:53.193281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:53.193255 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ca6972-5f95d6f8f9-pnbj6"] Nov 02 15:34:53.294093 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:53.294001 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w"] Nov 02 15:34:53.298420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:53.298390 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ca6972-754dd79d67-xtv4w"] Nov 02 15:34:54.459111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:54.459070 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c507e533-234e-46ac-b071-5718b372f7b6" path="/var/lib/kubelet/pods/c507e533-234e-46ac-b071-5718b372f7b6/volumes" Nov 02 15:34:54.459592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:54.459576 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e920348e-e97f-4d71-901e-af22e14ff51f" path="/var/lib/kubelet/pods/e920348e-e97f-4d71-901e-af22e14ff51f/volumes" Nov 02 15:34:57.157631 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:57.157593 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:58.212222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:58.212188 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:34:58.355688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:34:58.355638 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:02.560955 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:02.560925 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:07.161992 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.161938 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5"] Nov 02 15:35:07.162905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.162848 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" podUID="a0625832-5f69-4153-8ec0-fec6e8507405" containerName="maas-api" containerID="cri-o://b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0" gracePeriod=30 Nov 02 15:35:07.327961 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.327890 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:07.664204 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.664172 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:35:07.740066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.739962 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0625832-5f69-4153-8ec0-fec6e8507405-kube-api-access-k2t85\" (UniqueName: \"kubernetes.io/projected/a0625832-5f69-4153-8ec0-fec6e8507405-kube-api-access-k2t85\") pod \"a0625832-5f69-4153-8ec0-fec6e8507405\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " Nov 02 15:35:07.740066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.740040 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0625832-5f69-4153-8ec0-fec6e8507405-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625832-5f69-4153-8ec0-fec6e8507405-openshift-service-ca\") pod \"a0625832-5f69-4153-8ec0-fec6e8507405\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " Nov 02 15:35:07.740280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.740112 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls\") pod \"a0625832-5f69-4153-8ec0-fec6e8507405\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " Nov 02 15:35:07.740280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.740130 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-metrics-certs\") pod \"a0625832-5f69-4153-8ec0-fec6e8507405\" (UID: \"a0625832-5f69-4153-8ec0-fec6e8507405\") " Nov 02 15:35:07.740413 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.740388 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0625832-5f69-4153-8ec0-fec6e8507405-openshift-service-ca" pod "a0625832-5f69-4153-8ec0-fec6e8507405" (UID: "a0625832-5f69-4153-8ec0-fec6e8507405"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:07.740707 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.740680 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0625832-5f69-4153-8ec0-fec6e8507405-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:07.742238 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.742211 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-metrics-certs" pod "a0625832-5f69-4153-8ec0-fec6e8507405" (UID: "a0625832-5f69-4153-8ec0-fec6e8507405"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:35:07.742345 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.742291 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls" pod "a0625832-5f69-4153-8ec0-fec6e8507405" (UID: "a0625832-5f69-4153-8ec0-fec6e8507405"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:35:07.742400 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.742377 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0625832-5f69-4153-8ec0-fec6e8507405-kube-api-access-k2t85" pod "a0625832-5f69-4153-8ec0-fec6e8507405" (UID: "a0625832-5f69-4153-8ec0-fec6e8507405"). InnerVolumeSpecName "kube-api-access-k2t85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:07.870153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.858367 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2t85\" (UniqueName: \"kubernetes.io/projected/a0625832-5f69-4153-8ec0-fec6e8507405-kube-api-access-k2t85\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:07.870153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.858414 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:07.870153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:07.858432 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0625832-5f69-4153-8ec0-fec6e8507405-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:08.090669 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.090584 5433 generic.go:356] "Generic (PLEG): container finished" podID="a0625832-5f69-4153-8ec0-fec6e8507405" containerID="b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0" exitCode=0 Nov 02 15:35:08.090669 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.090633 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" Nov 02 15:35:08.090669 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.090650 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" event={"ID":"a0625832-5f69-4153-8ec0-fec6e8507405","Type":"ContainerDied","Data":"b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0"} Nov 02 15:35:08.090894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.090702 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5" event={"ID":"a0625832-5f69-4153-8ec0-fec6e8507405","Type":"ContainerDied","Data":"2ceed37ccabfca132220dc7e56d5c1692bf10b77d718c658b1ca38c57922f449"} Nov 02 15:35:08.090894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.090732 5433 scope.go:122] "RemoveContainer" containerID="b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0" Nov 02 15:35:08.102673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.102651 5433 scope.go:122] "RemoveContainer" containerID="b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0" Nov 02 15:35:08.102920 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:08.102899 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0\": container with ID starting with b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0 not found: ID does not exist" containerID="b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0" Nov 02 15:35:08.102984 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.102928 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0"} err="failed to get container status \"b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0\": rpc error: code = NotFound desc = could not find container \"b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0\": container with ID starting with b8eef037348be0d4dab4f0410c971f1ee69af8368177b0da8968d7c3e0f6b4a0 not found: ID does not exist" Nov 02 15:35:08.247556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.247519 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5"] Nov 02 15:35:08.305997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.304354 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-941182-858d698f67-76zp5"] Nov 02 15:35:08.458410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.458378 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0625832-5f69-4153-8ec0-fec6e8507405" path="/var/lib/kubelet/pods/a0625832-5f69-4153-8ec0-fec6e8507405/volumes" Nov 02 15:35:08.500198 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:08.500171 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:09.168928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:09.168869 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:09.480775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:09.480687 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:09.654203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:09.654167 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:10.136201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:10.136162 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:11.068772 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:11.068733 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:35:11.944386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:11.944353 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t"] Nov 02 15:35:12.095183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:12.095147 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7"] Nov 02 15:35:12.328541 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:12.328459 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:12.472937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:12.472907 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:13.693410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:13.693369 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:13.948749 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:13.948661 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:14.106127 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:14.106084 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:15.197317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:15.197288 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:16.945513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:16.945472 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" podUID="27ac0d90-7058-4c43-ba23-50c7456ca2be" containerName="payload-processing" containerID="cri-o://5b2b70bbe24d1136531d77ab3c24664a2a4dc22ca8069e7e8c0a5174932c220e" gracePeriod=25 Nov 02 15:35:17.028993 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.028948 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:17.096216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.096160 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" podUID="58bd2643-a1b3-4c60-924b-b0d1aba3ef04" containerName="payload-pre-processing" containerID="cri-o://d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676" gracePeriod=25 Nov 02 15:35:17.293132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.293031 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:17.297047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.296805 5433 generic.go:356] "Generic (PLEG): container finished" podID="27ac0d90-7058-4c43-ba23-50c7456ca2be" containerID="5b2b70bbe24d1136531d77ab3c24664a2a4dc22ca8069e7e8c0a5174932c220e" exitCode=0 Nov 02 15:35:17.297047 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.296874 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" event={"ID":"27ac0d90-7058-4c43-ba23-50c7456ca2be","Type":"ContainerDied","Data":"5b2b70bbe24d1136531d77ab3c24664a2a4dc22ca8069e7e8c0a5174932c220e"} Nov 02 15:35:17.641205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.640958 5433 scope.go:122] "RemoveContainer" containerID="0d4c7b30d0d0d32e1babfe8d93584c25427161245b283e0304554bcdea2334cc" Nov 02 15:35:17.645201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.644341 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:35:17.647767 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.647741 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb"] Nov 02 15:35:17.649209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.649174 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c507e533-234e-46ac-b071-5718b372f7b6" containerName="payload-pre-processing" Nov 02 15:35:17.649417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.649399 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c507e533-234e-46ac-b071-5718b372f7b6" containerName="payload-pre-processing" Nov 02 15:35:17.649567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.649551 5433 cpu_manager.go:414] "containerMap: removing container" podUID="27ac0d90-7058-4c43-ba23-50c7456ca2be" containerName="payload-processing" Nov 02 15:35:17.649668 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.649657 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ac0d90-7058-4c43-ba23-50c7456ca2be" containerName="payload-processing" Nov 02 15:35:17.649771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.649760 5433 cpu_manager.go:414] "containerMap: removing container" podUID="e920348e-e97f-4d71-901e-af22e14ff51f" containerName="payload-processing" Nov 02 15:35:17.649849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.649839 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e920348e-e97f-4d71-901e-af22e14ff51f" containerName="payload-processing" Nov 02 15:35:17.649938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.649928 5433 cpu_manager.go:414] "containerMap: removing container" podUID="a0625832-5f69-4153-8ec0-fec6e8507405" containerName="maas-api" Nov 02 15:35:17.650043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.650031 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0625832-5f69-4153-8ec0-fec6e8507405" containerName="maas-api" Nov 02 15:35:17.650356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.650337 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e920348e-e97f-4d71-901e-af22e14ff51f" containerName="payload-processing" Nov 02 15:35:17.650471 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.650459 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0625832-5f69-4153-8ec0-fec6e8507405" containerName="maas-api" Nov 02 15:35:17.650572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.650561 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c507e533-234e-46ac-b071-5718b372f7b6" containerName="payload-pre-processing" Nov 02 15:35:17.650656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.650645 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ac0d90-7058-4c43-ba23-50c7456ca2be" containerName="payload-processing" Nov 02 15:35:17.658160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.657252 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:17.680323 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.679410 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:17.704730 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.704694 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb"] Nov 02 15:35:17.740045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.737950 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2203c8-dockercfg-rz557\"" Nov 02 15:35:17.740045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.738880 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2203c8\"" Nov 02 15:35:17.767528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.765700 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ac0d90-7058-4c43-ba23-50c7456ca2be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ac0d90-7058-4c43-ba23-50c7456ca2be-plugins-config-volume\") pod \"27ac0d90-7058-4c43-ba23-50c7456ca2be\" (UID: \"27ac0d90-7058-4c43-ba23-50c7456ca2be\") " Nov 02 15:35:17.767528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.767430 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ac0d90-7058-4c43-ba23-50c7456ca2be-plugins-config-volume" pod "27ac0d90-7058-4c43-ba23-50c7456ca2be" (UID: "27ac0d90-7058-4c43-ba23-50c7456ca2be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:17.767528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.767517 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ac0d90-7058-4c43-ba23-50c7456ca2be-kube-api-access-hhf7c\" (UniqueName: \"kubernetes.io/projected/27ac0d90-7058-4c43-ba23-50c7456ca2be-kube-api-access-hhf7c\") pod \"27ac0d90-7058-4c43-ba23-50c7456ca2be\" (UID: \"27ac0d90-7058-4c43-ba23-50c7456ca2be\") " Nov 02 15:35:17.776368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.768073 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4hc\" (UniqueName: \"kubernetes.io/projected/6fc8b9d9-3252-48b6-9644-3107f357b196-kube-api-access-dl4hc\") pod \"payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb\" (UID: \"6fc8b9d9-3252-48b6-9644-3107f357b196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:17.776368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.768191 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8b9d9-3252-48b6-9644-3107f357b196-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb\" (UID: \"6fc8b9d9-3252-48b6-9644-3107f357b196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:17.776368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.768341 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ac0d90-7058-4c43-ba23-50c7456ca2be-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:17.779042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.778965 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ac0d90-7058-4c43-ba23-50c7456ca2be-kube-api-access-hhf7c" pod "27ac0d90-7058-4c43-ba23-50c7456ca2be" (UID: "27ac0d90-7058-4c43-ba23-50c7456ca2be"). InnerVolumeSpecName "kube-api-access-hhf7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:17.816175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.816153 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:35:17.833035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.832969 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:17.870295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.870210 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4hc\" (UniqueName: \"kubernetes.io/projected/6fc8b9d9-3252-48b6-9644-3107f357b196-kube-api-access-dl4hc\") pod \"payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb\" (UID: \"6fc8b9d9-3252-48b6-9644-3107f357b196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:17.870295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.870285 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8b9d9-3252-48b6-9644-3107f357b196-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb\" (UID: \"6fc8b9d9-3252-48b6-9644-3107f357b196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:17.870526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.870394 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhf7c\" (UniqueName: \"kubernetes.io/projected/27ac0d90-7058-4c43-ba23-50c7456ca2be-kube-api-access-hhf7c\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:17.871149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.871125 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8b9d9-3252-48b6-9644-3107f357b196-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb\" (UID: \"6fc8b9d9-3252-48b6-9644-3107f357b196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:17.933012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.932959 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4hc\" (UniqueName: \"kubernetes.io/projected/6fc8b9d9-3252-48b6-9644-3107f357b196-kube-api-access-dl4hc\") pod \"payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb\" (UID: \"6fc8b9d9-3252-48b6-9644-3107f357b196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:17.971555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.971521 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-kube-api-access-r9j8q\" (UniqueName: \"kubernetes.io/projected/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-kube-api-access-r9j8q\") pod \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\" (UID: \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\") " Nov 02 15:35:17.972201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.972176 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-plugins-config-volume\") pod \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\" (UID: \"58bd2643-a1b3-4c60-924b-b0d1aba3ef04\") " Nov 02 15:35:17.972737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.972709 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-plugins-config-volume" pod "58bd2643-a1b3-4c60-924b-b0d1aba3ef04" (UID: "58bd2643-a1b3-4c60-924b-b0d1aba3ef04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:17.973617 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.973577 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:17.974533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:17.974508 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-kube-api-access-r9j8q" pod "58bd2643-a1b3-4c60-924b-b0d1aba3ef04" (UID: "58bd2643-a1b3-4c60-924b-b0d1aba3ef04"). InnerVolumeSpecName "kube-api-access-r9j8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:18.010004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.009959 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd"] Nov 02 15:35:18.010484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.010454 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" containerID="cri-o://f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d" gracePeriod=30 Nov 02 15:35:18.014746 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.014718 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:18.048560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.043002 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:18.075714 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.075675 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9j8q\" (UniqueName: \"kubernetes.io/projected/58bd2643-a1b3-4c60-924b-b0d1aba3ef04-kube-api-access-r9j8q\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:18.336913 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.334247 5433 generic.go:356] "Generic (PLEG): container finished" podID="58bd2643-a1b3-4c60-924b-b0d1aba3ef04" containerID="d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676" exitCode=0 Nov 02 15:35:18.336913 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.334519 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" event={"ID":"58bd2643-a1b3-4c60-924b-b0d1aba3ef04","Type":"ContainerDied","Data":"d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676"} Nov 02 15:35:18.336913 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.334597 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" event={"ID":"58bd2643-a1b3-4c60-924b-b0d1aba3ef04","Type":"ContainerDied","Data":"8c01aeb11aa4144d928cb45f8aa27355d019283ef4b5737b33cf58dc609b8392"} Nov 02 15:35:18.336913 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.334635 5433 scope.go:122] "RemoveContainer" containerID="d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676" Nov 02 15:35:18.336913 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.334868 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7" Nov 02 15:35:18.343864 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.343831 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" event={"ID":"27ac0d90-7058-4c43-ba23-50c7456ca2be","Type":"ContainerDied","Data":"61ce90c08d74e4fe8578491364bcfb401c66b8c867f0f9d1d30e9b2297187d91"} Nov 02 15:35:18.344020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.343968 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t" Nov 02 15:35:18.355428 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.355353 5433 scope.go:122] "RemoveContainer" containerID="d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676" Nov 02 15:35:18.355921 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:18.355891 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676\": container with ID starting with d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676 not found: ID does not exist" containerID="d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676" Nov 02 15:35:18.356032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.355948 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676"} err="failed to get container status \"d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676\": rpc error: code = NotFound desc = could not find container \"d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676\": container with ID starting with d62a1ea471cfde1678fb53a0059efbd11f6aa8c72a817fba763ae5894bb57676 not found: ID does not exist" Nov 02 15:35:18.356032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.355986 5433 scope.go:122] "RemoveContainer" containerID="5b2b70bbe24d1136531d77ab3c24664a2a4dc22ca8069e7e8c0a5174932c220e" Nov 02 15:35:18.507074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.506999 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7"] Nov 02 15:35:18.524014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.523964 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1c449c-56bc44cb5f-rpqf7"] Nov 02 15:35:18.625443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.625411 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb"] Nov 02 15:35:18.626012 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:35:18.625967 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc8b9d9_3252_48b6_9644_3107f357b196.slice/crio-97a7fbafcbdbfcd0ced4922e45d415a9e32119eabca3d07e9137f0510749bd37 WatchSource:0}: Error finding container 97a7fbafcbdbfcd0ced4922e45d415a9e32119eabca3d07e9137f0510749bd37: Status 404 returned error can't find the container with id 97a7fbafcbdbfcd0ced4922e45d415a9e32119eabca3d07e9137f0510749bd37 Nov 02 15:35:18.630948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.630918 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t"] Nov 02 15:35:18.635834 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:18.635810 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1c449c-69b9c979cc-wp96t"] Nov 02 15:35:19.354486 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:19.354447 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" event={"ID":"6fc8b9d9-3252-48b6-9644-3107f357b196","Type":"ContainerStarted","Data":"560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3"} Nov 02 15:35:19.354943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:19.354510 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" event={"ID":"6fc8b9d9-3252-48b6-9644-3107f357b196","Type":"ContainerStarted","Data":"97a7fbafcbdbfcd0ced4922e45d415a9e32119eabca3d07e9137f0510749bd37"} Nov 02 15:35:19.354943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:19.354535 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:19.467666 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:19.467602 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" podStartSLOduration=2.467584091 podStartE2EDuration="2.467584091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:19.46437646 +0000 UTC m=+2223.137579656" watchObservedRunningTime="2026-11-02 15:35:19.467584091 +0000 UTC m=+2223.140787288" Nov 02 15:35:20.457760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.457729 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ac0d90-7058-4c43-ba23-50c7456ca2be" path="/var/lib/kubelet/pods/27ac0d90-7058-4c43-ba23-50c7456ca2be/volumes" Nov 02 15:35:20.458228 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.458213 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58bd2643-a1b3-4c60-924b-b0d1aba3ef04" path="/var/lib/kubelet/pods/58bd2643-a1b3-4c60-924b-b0d1aba3ef04/volumes" Nov 02 15:35:20.486765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.486737 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj"] Nov 02 15:35:20.487464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.487442 5433 cpu_manager.go:414] "containerMap: removing container" podUID="58bd2643-a1b3-4c60-924b-b0d1aba3ef04" containerName="payload-pre-processing" Nov 02 15:35:20.487464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.487462 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bd2643-a1b3-4c60-924b-b0d1aba3ef04" containerName="payload-pre-processing" Nov 02 15:35:20.487639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.487627 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bd2643-a1b3-4c60-924b-b0d1aba3ef04" containerName="payload-pre-processing" Nov 02 15:35:20.492546 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.492528 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.543881 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.543854 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5aeac88e-serving-cert\"" Nov 02 15:35:20.547664 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.547633 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj"] Nov 02 15:35:20.634120 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.634088 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trdjh\" (UniqueName: \"kubernetes.io/projected/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-kube-api-access-trdjh\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.634310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.634288 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-openshift-service-ca\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.634381 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.634354 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-metrics-certs\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.634479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.634459 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-maas-api-tls\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.688299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.688267 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:20.765066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.758576 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trdjh\" (UniqueName: \"kubernetes.io/projected/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-kube-api-access-trdjh\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.765066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.758782 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-openshift-service-ca\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.765066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.758878 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-metrics-certs\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.765066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.758940 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-maas-api-tls\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.784996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.775048 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-openshift-service-ca\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.803016 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.800653 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-metrics-certs\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.803016 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.801024 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-maas-api-tls\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.831508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.831476 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trdjh\" (UniqueName: \"kubernetes.io/projected/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-kube-api-access-trdjh\") pod \"maas-api-e2e-mt-5aeac88e-77967d9754-k4snj\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:20.865336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:20.865305 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:21.009303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:21.009270 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:21.107204 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:21.107120 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:21.483564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:21.483537 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj"] Nov 02 15:35:21.485225 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:35:21.485198 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7ecef6_fb93_4411_b97c_3bdfe202e47f.slice/crio-3292ec99b251eefa7aa1bd5915a59deadd8a6d448579c0fc664f0306bf5db550 WatchSource:0}: Error finding container 3292ec99b251eefa7aa1bd5915a59deadd8a6d448579c0fc664f0306bf5db550: Status 404 returned error can't find the container with id 3292ec99b251eefa7aa1bd5915a59deadd8a6d448579c0fc664f0306bf5db550 Nov 02 15:35:21.487049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:21.487028 5433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:35:22.378391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:22.378354 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" event={"ID":"2e7ecef6-fb93-4411-b97c-3bdfe202e47f","Type":"ContainerStarted","Data":"3292ec99b251eefa7aa1bd5915a59deadd8a6d448579c0fc664f0306bf5db550"} Nov 02 15:35:23.297549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.297487 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:35:23.300943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.300922 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230df24c-62fc-42d9-a0a6-480d2e142269-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230df24c-62fc-42d9-a0a6-480d2e142269-istiod-ca-cert\") pod \"230df24c-62fc-42d9-a0a6-480d2e142269\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " Nov 02 15:35:23.301051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.300961 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-credential-socket\") pod \"230df24c-62fc-42d9-a0a6-480d2e142269\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " Nov 02 15:35:23.301051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301042 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-istio-token\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-istio-token\") pod \"230df24c-62fc-42d9-a0a6-480d2e142269\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " Nov 02 15:35:23.301154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301068 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-data\") pod \"230df24c-62fc-42d9-a0a6-480d2e142269\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " Nov 02 15:35:23.301154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301086 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-kube-api-access-rcd75\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-kube-api-access-rcd75\") pod \"230df24c-62fc-42d9-a0a6-480d2e142269\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " Nov 02 15:35:23.301264 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301155 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-envoy\") pod \"230df24c-62fc-42d9-a0a6-480d2e142269\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " Nov 02 15:35:23.301264 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301208 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-socket\") pod \"230df24c-62fc-42d9-a0a6-480d2e142269\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " Nov 02 15:35:23.301358 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301268 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-certs\") pod \"230df24c-62fc-42d9-a0a6-480d2e142269\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " Nov 02 15:35:23.301358 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301311 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/230df24c-62fc-42d9-a0a6-480d2e142269-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230df24c-62fc-42d9-a0a6-480d2e142269-istio-podinfo\") pod \"230df24c-62fc-42d9-a0a6-480d2e142269\" (UID: \"230df24c-62fc-42d9-a0a6-480d2e142269\") " Nov 02 15:35:23.301461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301404 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230df24c-62fc-42d9-a0a6-480d2e142269-istiod-ca-cert" pod "230df24c-62fc-42d9-a0a6-480d2e142269" (UID: "230df24c-62fc-42d9-a0a6-480d2e142269"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:23.301461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301406 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-credential-socket" pod "230df24c-62fc-42d9-a0a6-480d2e142269" (UID: "230df24c-62fc-42d9-a0a6-480d2e142269"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:23.301461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301435 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-data" pod "230df24c-62fc-42d9-a0a6-480d2e142269" (UID: "230df24c-62fc-42d9-a0a6-480d2e142269"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:23.301608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301515 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-socket" pod "230df24c-62fc-42d9-a0a6-480d2e142269" (UID: "230df24c-62fc-42d9-a0a6-480d2e142269"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:23.301608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301595 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-certs" pod "230df24c-62fc-42d9-a0a6-480d2e142269" (UID: "230df24c-62fc-42d9-a0a6-480d2e142269"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:23.301915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301896 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:23.302031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301917 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:23.302031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301931 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230df24c-62fc-42d9-a0a6-480d2e142269-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:23.302031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301944 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:23.302031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.301959 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:23.303548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.303525 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-envoy" pod "230df24c-62fc-42d9-a0a6-480d2e142269" (UID: "230df24c-62fc-42d9-a0a6-480d2e142269"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:35:23.303548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.303530 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/230df24c-62fc-42d9-a0a6-480d2e142269-istio-podinfo" pod "230df24c-62fc-42d9-a0a6-480d2e142269" (UID: "230df24c-62fc-42d9-a0a6-480d2e142269"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:35:23.303849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.303828 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-istio-token" pod "230df24c-62fc-42d9-a0a6-480d2e142269" (UID: "230df24c-62fc-42d9-a0a6-480d2e142269"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:23.303933 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.303855 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-kube-api-access-rcd75" pod "230df24c-62fc-42d9-a0a6-480d2e142269" (UID: "230df24c-62fc-42d9-a0a6-480d2e142269"). InnerVolumeSpecName "kube-api-access-rcd75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:23.386628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.386587 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" event={"ID":"2e7ecef6-fb93-4411-b97c-3bdfe202e47f","Type":"ContainerStarted","Data":"d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12"} Nov 02 15:35:23.386803 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.386667 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:23.388393 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.388366 5433 generic.go:356] "Generic (PLEG): container finished" podID="230df24c-62fc-42d9-a0a6-480d2e142269" containerID="f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d" exitCode=0 Nov 02 15:35:23.388522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.388408 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" Nov 02 15:35:23.388522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.388452 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" event={"ID":"230df24c-62fc-42d9-a0a6-480d2e142269","Type":"ContainerDied","Data":"f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d"} Nov 02 15:35:23.388522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.388504 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd" event={"ID":"230df24c-62fc-42d9-a0a6-480d2e142269","Type":"ContainerDied","Data":"67b585b96fc118073aa3d41a7882957b4167c243131d2292b2c90416c9d6f620"} Nov 02 15:35:23.388641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.388526 5433 scope.go:122] "RemoveContainer" containerID="f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d" Nov 02 15:35:23.399708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.399687 5433 scope.go:122] "RemoveContainer" containerID="f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b" Nov 02 15:35:23.403078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.403054 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230df24c-62fc-42d9-a0a6-480d2e142269-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:23.403078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.403077 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230df24c-62fc-42d9-a0a6-480d2e142269-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:23.403222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.403087 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:23.403222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.403099 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcd75\" (UniqueName: \"kubernetes.io/projected/230df24c-62fc-42d9-a0a6-480d2e142269-kube-api-access-rcd75\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:23.409042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.409027 5433 scope.go:122] "RemoveContainer" containerID="f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d" Nov 02 15:35:23.409306 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:23.409290 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d\": container with ID starting with f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d not found: ID does not exist" containerID="f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d" Nov 02 15:35:23.409342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.409313 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d"} err="failed to get container status \"f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d\": rpc error: code = NotFound desc = could not find container \"f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d\": container with ID starting with f469c08359621a5353100d31bae78c8b81b30b9ba936ca6ffb46b9cd05706c6d not found: ID does not exist" Nov 02 15:35:23.409342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.409331 5433 scope.go:122] "RemoveContainer" containerID="f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b" Nov 02 15:35:23.409574 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:23.409556 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b\": container with ID starting with f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b not found: ID does not exist" containerID="f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b" Nov 02 15:35:23.409612 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.409580 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b"} err="failed to get container status \"f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b\": rpc error: code = NotFound desc = could not find container \"f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b\": container with ID starting with f8abb620413f371e492ec16abeefa7fdb7ae529dff1eedcef4e4bf886f81b93b not found: ID does not exist" Nov 02 15:35:23.526036 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.523414 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" podStartSLOduration=2.536463054 podStartE2EDuration="3.523395177s" totalImagesPullingTime="986.932123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:23.521477373 +0000 UTC m=+2227.194680567" watchObservedRunningTime="2026-11-02 15:35:23.523395177 +0000 UTC m=+2227.196598373" Nov 02 15:35:23.628500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.628469 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd"] Nov 02 15:35:23.637673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:23.634715 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-941182-openshift-default-6d59886dfc-cfmgd"] Nov 02 15:35:24.457599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:24.457565 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" path="/var/lib/kubelet/pods/230df24c-62fc-42d9-a0a6-480d2e142269/volumes" Nov 02 15:35:26.079312 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:26.079277 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:28.693820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:28.693766 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:29.402038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:29.402004 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:35:30.365233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:30.365205 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:35:30.457855 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:30.457826 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg"] Nov 02 15:35:30.712281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:30.712250 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:30.854546 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:30.854502 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:31.179130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:31.179096 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:35.458850 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:35.458798 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" podUID="19a3bbeb-074e-48eb-99e3-4186eee1155b" containerName="payload-pre-processing" containerID="cri-o://81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57" gracePeriod=25 Nov 02 15:35:35.736045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:35.734806 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:35:35.888478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:35.888449 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a3bbeb-074e-48eb-99e3-4186eee1155b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a3bbeb-074e-48eb-99e3-4186eee1155b-plugins-config-volume\") pod \"19a3bbeb-074e-48eb-99e3-4186eee1155b\" (UID: \"19a3bbeb-074e-48eb-99e3-4186eee1155b\") " Nov 02 15:35:35.888683 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:35.888615 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a3bbeb-074e-48eb-99e3-4186eee1155b-kube-api-access-l99kj\" (UniqueName: \"kubernetes.io/projected/19a3bbeb-074e-48eb-99e3-4186eee1155b-kube-api-access-l99kj\") pod \"19a3bbeb-074e-48eb-99e3-4186eee1155b\" (UID: \"19a3bbeb-074e-48eb-99e3-4186eee1155b\") " Nov 02 15:35:35.888932 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:35.888903 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a3bbeb-074e-48eb-99e3-4186eee1155b-plugins-config-volume" pod "19a3bbeb-074e-48eb-99e3-4186eee1155b" (UID: "19a3bbeb-074e-48eb-99e3-4186eee1155b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:35.889177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:35.889157 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a3bbeb-074e-48eb-99e3-4186eee1155b-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:35.891152 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:35.891126 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a3bbeb-074e-48eb-99e3-4186eee1155b-kube-api-access-l99kj" pod "19a3bbeb-074e-48eb-99e3-4186eee1155b" (UID: "19a3bbeb-074e-48eb-99e3-4186eee1155b"). InnerVolumeSpecName "kube-api-access-l99kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:35.990900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:35.990826 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l99kj\" (UniqueName: \"kubernetes.io/projected/19a3bbeb-074e-48eb-99e3-4186eee1155b-kube-api-access-l99kj\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:36.265074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.264957 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:36.479788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.479757 5433 generic.go:356] "Generic (PLEG): container finished" podID="19a3bbeb-074e-48eb-99e3-4186eee1155b" containerID="81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57" exitCode=0 Nov 02 15:35:36.480207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.479806 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" Nov 02 15:35:36.480207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.479838 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" event={"ID":"19a3bbeb-074e-48eb-99e3-4186eee1155b","Type":"ContainerDied","Data":"81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57"} Nov 02 15:35:36.480207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.479887 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg" event={"ID":"19a3bbeb-074e-48eb-99e3-4186eee1155b","Type":"ContainerDied","Data":"9a0c1bdd4c12cbc8ff04c2753505d6a4576505d61ed5f1e2e99d1d310ae0ba4e"} Nov 02 15:35:36.480207 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.479909 5433 scope.go:122] "RemoveContainer" containerID="81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57" Nov 02 15:35:36.491855 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.491834 5433 scope.go:122] "RemoveContainer" containerID="81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57" Nov 02 15:35:36.492160 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:36.492134 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57\": container with ID starting with 81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57 not found: ID does not exist" containerID="81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57" Nov 02 15:35:36.492272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.492171 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57"} err="failed to get container status \"81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57\": rpc error: code = NotFound desc = could not find container \"81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57\": container with ID starting with 81921382e62eeed58dff91a2ecc85095cf43ff27f38059fae7514466df071b57 not found: ID does not exist" Nov 02 15:35:36.633910 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.633873 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg"] Nov 02 15:35:36.679915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:36.679878 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-fb80ca-7b984b96ff-78jqg"] Nov 02 15:35:38.457187 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:38.457153 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a3bbeb-074e-48eb-99e3-4186eee1155b" path="/var/lib/kubelet/pods/19a3bbeb-074e-48eb-99e3-4186eee1155b/volumes" Nov 02 15:35:41.016454 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:41.016406 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:35:41.144597 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:41.144563 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:41.315606 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:41.315501 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:41.461206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:41.461167 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:46.540957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:46.540910 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:50.862093 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:50.862059 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt"] Nov 02 15:35:50.862492 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:50.862265 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" podUID="1b75064f-9c31-48d5-b496-84582cb6c5e0" containerName="maas-api" containerID="cri-o://e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39" gracePeriod=30 Nov 02 15:35:50.963319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:50.963262 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:51.207778 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.207751 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:35:51.335096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.315288 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls\") pod \"1b75064f-9c31-48d5-b496-84582cb6c5e0\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " Nov 02 15:35:51.335096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.315346 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b75064f-9c31-48d5-b496-84582cb6c5e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75064f-9c31-48d5-b496-84582cb6c5e0-openshift-service-ca\") pod \"1b75064f-9c31-48d5-b496-84582cb6c5e0\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " Nov 02 15:35:51.335096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.315618 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-metrics-certs\") pod \"1b75064f-9c31-48d5-b496-84582cb6c5e0\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " Nov 02 15:35:51.335096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.315661 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b75064f-9c31-48d5-b496-84582cb6c5e0-kube-api-access-lmqst\" (UniqueName: \"kubernetes.io/projected/1b75064f-9c31-48d5-b496-84582cb6c5e0-kube-api-access-lmqst\") pod \"1b75064f-9c31-48d5-b496-84582cb6c5e0\" (UID: \"1b75064f-9c31-48d5-b496-84582cb6c5e0\") " Nov 02 15:35:51.340189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.340154 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b75064f-9c31-48d5-b496-84582cb6c5e0-openshift-service-ca" pod "1b75064f-9c31-48d5-b496-84582cb6c5e0" (UID: "1b75064f-9c31-48d5-b496-84582cb6c5e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:51.356163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.351232 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b75064f-9c31-48d5-b496-84582cb6c5e0-kube-api-access-lmqst" pod "1b75064f-9c31-48d5-b496-84582cb6c5e0" (UID: "1b75064f-9c31-48d5-b496-84582cb6c5e0"). InnerVolumeSpecName "kube-api-access-lmqst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:51.356163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.351374 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-metrics-certs" pod "1b75064f-9c31-48d5-b496-84582cb6c5e0" (UID: "1b75064f-9c31-48d5-b496-84582cb6c5e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:35:51.356163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.355836 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls" pod "1b75064f-9c31-48d5-b496-84582cb6c5e0" (UID: "1b75064f-9c31-48d5-b496-84582cb6c5e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:35:51.417811 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.417776 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:51.417811 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.417811 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmqst\" (UniqueName: \"kubernetes.io/projected/1b75064f-9c31-48d5-b496-84582cb6c5e0-kube-api-access-lmqst\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:51.418057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.417824 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b75064f-9c31-48d5-b496-84582cb6c5e0-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:51.418057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.417833 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75064f-9c31-48d5-b496-84582cb6c5e0-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:51.592907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.592814 5433 generic.go:356] "Generic (PLEG): container finished" podID="1b75064f-9c31-48d5-b496-84582cb6c5e0" containerID="e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39" exitCode=0 Nov 02 15:35:51.592907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.592866 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" Nov 02 15:35:51.593146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.592901 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" event={"ID":"1b75064f-9c31-48d5-b496-84582cb6c5e0","Type":"ContainerDied","Data":"e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39"} Nov 02 15:35:51.593146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.593008 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt" event={"ID":"1b75064f-9c31-48d5-b496-84582cb6c5e0","Type":"ContainerDied","Data":"8ea39c2886cfc7eef54c7d24ccfcd827e4ea7e66ad4b3d403cc4ce075ecca677"} Nov 02 15:35:51.593146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.593055 5433 scope.go:122] "RemoveContainer" containerID="e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39" Nov 02 15:35:51.604351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.604311 5433 scope.go:122] "RemoveContainer" containerID="e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39" Nov 02 15:35:51.604616 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:51.604594 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39\": container with ID starting with e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39 not found: ID does not exist" containerID="e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39" Nov 02 15:35:51.604703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.604631 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39"} err="failed to get container status \"e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39\": rpc error: code = NotFound desc = could not find container \"e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39\": container with ID starting with e0029bab9cc94890ba6a8ea71756dc457bffafd39589b4c394e79823115dbd39 not found: ID does not exist" Nov 02 15:35:51.748132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.748094 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt"] Nov 02 15:35:51.751175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:51.751150 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-ca6972-74b5c54889-ccftt"] Nov 02 15:35:52.457951 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.457912 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b75064f-9c31-48d5-b496-84582cb6c5e0" path="/var/lib/kubelet/pods/1b75064f-9c31-48d5-b496-84582cb6c5e0/volumes" Nov 02 15:35:52.588771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.588730 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29"] Nov 02 15:35:52.589541 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589515 5433 cpu_manager.go:414] "containerMap: removing container" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" Nov 02 15:35:52.589541 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589539 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" Nov 02 15:35:52.589736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589570 5433 cpu_manager.go:414] "containerMap: removing container" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" Nov 02 15:35:52.589736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589577 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" Nov 02 15:35:52.589736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589605 5433 cpu_manager.go:414] "containerMap: removing container" podUID="1b75064f-9c31-48d5-b496-84582cb6c5e0" containerName="maas-api" Nov 02 15:35:52.589736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589612 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b75064f-9c31-48d5-b496-84582cb6c5e0" containerName="maas-api" Nov 02 15:35:52.589736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589627 5433 cpu_manager.go:414] "containerMap: removing container" podUID="19a3bbeb-074e-48eb-99e3-4186eee1155b" containerName="payload-pre-processing" Nov 02 15:35:52.589736 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589634 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a3bbeb-074e-48eb-99e3-4186eee1155b" containerName="payload-pre-processing" Nov 02 15:35:52.590056 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589838 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" Nov 02 15:35:52.590056 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589857 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b75064f-9c31-48d5-b496-84582cb6c5e0" containerName="maas-api" Nov 02 15:35:52.590056 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.589868 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a3bbeb-074e-48eb-99e3-4186eee1155b" containerName="payload-pre-processing" Nov 02 15:35:52.594915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.594889 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.606314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.606289 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:52.643134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.643101 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md9g4\" (UniqueName: \"kubernetes.io/projected/48a59200-a748-4b25-b837-0eefea63069a-kube-api-access-md9g4\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.643289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.643213 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.643289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.643277 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-metrics-certs\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.643368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.643311 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a59200-a748-4b25-b837-0eefea63069a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.694168 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.694132 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-822f77-serving-cert\"" Nov 02 15:35:52.697469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.697444 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29"] Nov 02 15:35:52.744642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.744542 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md9g4\" (UniqueName: \"kubernetes.io/projected/48a59200-a748-4b25-b837-0eefea63069a-kube-api-access-md9g4\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.744953 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.744925 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.745127 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.745045 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-metrics-certs\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.745127 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.745111 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a59200-a748-4b25-b837-0eefea63069a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.745637 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:52.745613 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-serving-cert: secret "maas-api-e2e-inf-a-822f77-serving-cert" not found Nov 02 15:35:52.745774 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:35:52.745730 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls podName:48a59200-a748-4b25-b837-0eefea63069a nodeName:}" failed. No retries permitted until 2026-11-02 15:35:53.245707015 +0000 UTC m=+2256.918910205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls") pod "maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" (UID: "48a59200-a748-4b25-b837-0eefea63069a") : secret "maas-api-e2e-inf-a-822f77-serving-cert" not found Nov 02 15:35:52.745948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.745919 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a59200-a748-4b25-b837-0eefea63069a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.748411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.748385 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-metrics-certs\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.802359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.802325 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md9g4\" (UniqueName: \"kubernetes.io/projected/48a59200-a748-4b25-b837-0eefea63069a-kube-api-access-md9g4\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:52.835174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:52.835138 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:53.034673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:53.034583 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:53.192735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:53.192704 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:53.280727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:53.280692 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:53.283195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:53.283170 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls\") pod \"maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:53.510095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:53.510056 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:53.544465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:53.544429 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:53.891416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:53.890626 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29"] Nov 02 15:35:53.892075 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:35:53.892042 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a59200_a748_4b25_b837_0eefea63069a.slice/crio-a2cd9a26ad375b9a85c16906e3e77ddd82515062a279ca1ad5f7198b54466f1a WatchSource:0}: Error finding container a2cd9a26ad375b9a85c16906e3e77ddd82515062a279ca1ad5f7198b54466f1a: Status 404 returned error can't find the container with id a2cd9a26ad375b9a85c16906e3e77ddd82515062a279ca1ad5f7198b54466f1a Nov 02 15:35:54.615734 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:54.615700 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" event={"ID":"48a59200-a748-4b25-b837-0eefea63069a","Type":"ContainerStarted","Data":"a2cd9a26ad375b9a85c16906e3e77ddd82515062a279ca1ad5f7198b54466f1a"} Nov 02 15:35:55.623440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:55.623402 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" event={"ID":"48a59200-a748-4b25-b837-0eefea63069a","Type":"ContainerStarted","Data":"4b9ec91a307f26d6daf7a0d9f9115023d5355011ef7bcb89f9af70516cf5d3a8"} Nov 02 15:35:55.623916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:55.623495 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:35:55.737033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:55.736966 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" podStartSLOduration=2.879384954 podStartE2EDuration="3.736951529s" totalImagesPullingTime="857.566575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:55.734190328 +0000 UTC m=+2259.407393522" watchObservedRunningTime="2026-11-02 15:35:55.736951529 +0000 UTC m=+2259.410154724" Nov 02 15:35:56.196424 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:56.196388 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:35:58.545692 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:35:58.545632 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:00.011006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:00.010924 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:00.162013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:00.161931 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:00.392188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:00.392154 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj"] Nov 02 15:36:00.392446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:00.392419 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" containerID="cri-o://cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a" gracePeriod=30 Nov 02 15:36:00.403179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:00.403139 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:01.637266 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:01.637232 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:36:04.160469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:04.160436 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:05.672250 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.672190 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:36:05.692069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.692039 5433 generic.go:356] "Generic (PLEG): container finished" podID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerID="cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a" exitCode=0 Nov 02 15:36:05.692230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.692088 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" Nov 02 15:36:05.692230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.692124 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" event={"ID":"0516903c-1765-4aaf-9886-47e2ebfe0117","Type":"ContainerDied","Data":"cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a"} Nov 02 15:36:05.692230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.692186 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj" event={"ID":"0516903c-1765-4aaf-9886-47e2ebfe0117","Type":"ContainerDied","Data":"542ca803e91ec5d136d0dad7eee71e75949f1afcc2f3d6b5b15e60cd3d297a82"} Nov 02 15:36:05.692230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.692217 5433 scope.go:122] "RemoveContainer" containerID="cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a" Nov 02 15:36:05.701815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.701799 5433 scope.go:122] "RemoveContainer" containerID="f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382" Nov 02 15:36:05.711095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.711073 5433 scope.go:122] "RemoveContainer" containerID="cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a" Nov 02 15:36:05.711395 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:05.711375 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a\": container with ID starting with cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a not found: ID does not exist" containerID="cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a" Nov 02 15:36:05.711490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.711404 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a"} err="failed to get container status \"cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a\": rpc error: code = NotFound desc = could not find container \"cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a\": container with ID starting with cedc8ff4e81fdb389f9fe8eb1d1315fe91a6db58fea6815d524b6b34eb42792a not found: ID does not exist" Nov 02 15:36:05.711490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.711423 5433 scope.go:122] "RemoveContainer" containerID="f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382" Nov 02 15:36:05.711708 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:05.711691 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382\": container with ID starting with f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382 not found: ID does not exist" containerID="f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382" Nov 02 15:36:05.711756 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.711713 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382"} err="failed to get container status \"f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382\": rpc error: code = NotFound desc = could not find container \"f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382\": container with ID starting with f9ad51448501d266a2a7789812d486ea8b5ac1d6ffd1cab49bde93a666628382 not found: ID does not exist" Nov 02 15:36:05.767294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767269 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-credential-socket\") pod \"0516903c-1765-4aaf-9886-47e2ebfe0117\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " Nov 02 15:36:05.767494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767304 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-socket\") pod \"0516903c-1765-4aaf-9886-47e2ebfe0117\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " Nov 02 15:36:05.767494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767358 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-data\") pod \"0516903c-1765-4aaf-9886-47e2ebfe0117\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " Nov 02 15:36:05.767494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767395 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-certs\") pod \"0516903c-1765-4aaf-9886-47e2ebfe0117\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " Nov 02 15:36:05.767494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767442 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-podinfo\") pod \"0516903c-1765-4aaf-9886-47e2ebfe0117\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " Nov 02 15:36:05.767708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767499 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-token\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-token\") pod \"0516903c-1765-4aaf-9886-47e2ebfe0117\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " Nov 02 15:36:05.767708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767555 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0516903c-1765-4aaf-9886-47e2ebfe0117-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0516903c-1765-4aaf-9886-47e2ebfe0117-istiod-ca-cert\") pod \"0516903c-1765-4aaf-9886-47e2ebfe0117\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " Nov 02 15:36:05.767708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767561 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-credential-socket" pod "0516903c-1765-4aaf-9886-47e2ebfe0117" (UID: "0516903c-1765-4aaf-9886-47e2ebfe0117"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:05.767708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767584 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-kube-api-access-dscd9\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-kube-api-access-dscd9\") pod \"0516903c-1765-4aaf-9886-47e2ebfe0117\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " Nov 02 15:36:05.767708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767630 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-envoy\") pod \"0516903c-1765-4aaf-9886-47e2ebfe0117\" (UID: \"0516903c-1765-4aaf-9886-47e2ebfe0117\") " Nov 02 15:36:05.767960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767792 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-certs" pod "0516903c-1765-4aaf-9886-47e2ebfe0117" (UID: "0516903c-1765-4aaf-9886-47e2ebfe0117"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:05.767960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.767843 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-socket" pod "0516903c-1765-4aaf-9886-47e2ebfe0117" (UID: "0516903c-1765-4aaf-9886-47e2ebfe0117"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:05.768105 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.768035 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-data" pod "0516903c-1765-4aaf-9886-47e2ebfe0117" (UID: "0516903c-1765-4aaf-9886-47e2ebfe0117"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:05.768359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.768333 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.768463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.768362 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.768463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.768376 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.768463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.768392 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.768731 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.768711 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0516903c-1765-4aaf-9886-47e2ebfe0117-istiod-ca-cert" pod "0516903c-1765-4aaf-9886-47e2ebfe0117" (UID: "0516903c-1765-4aaf-9886-47e2ebfe0117"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:05.769930 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.769902 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-envoy" pod "0516903c-1765-4aaf-9886-47e2ebfe0117" (UID: "0516903c-1765-4aaf-9886-47e2ebfe0117"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:05.770156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.770138 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-podinfo" pod "0516903c-1765-4aaf-9886-47e2ebfe0117" (UID: "0516903c-1765-4aaf-9886-47e2ebfe0117"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:36:05.770236 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.770193 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-token" pod "0516903c-1765-4aaf-9886-47e2ebfe0117" (UID: "0516903c-1765-4aaf-9886-47e2ebfe0117"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:05.770464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.770435 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-kube-api-access-dscd9" pod "0516903c-1765-4aaf-9886-47e2ebfe0117" (UID: "0516903c-1765-4aaf-9886-47e2ebfe0117"). InnerVolumeSpecName "kube-api-access-dscd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:05.871173 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.871138 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.871173 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.871173 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.871358 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.871187 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0516903c-1765-4aaf-9886-47e2ebfe0117-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.871358 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.871203 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dscd9\" (UniqueName: \"kubernetes.io/projected/0516903c-1765-4aaf-9886-47e2ebfe0117-kube-api-access-dscd9\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.871358 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:05.871217 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0516903c-1765-4aaf-9886-47e2ebfe0117-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:06.108822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:06.108790 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj"] Nov 02 15:36:06.113387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:06.113360 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ca6972-openshift-default-5cf464d5c-tqphj"] Nov 02 15:36:06.457177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:06.457145 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" path="/var/lib/kubelet/pods/0516903c-1765-4aaf-9886-47e2ebfe0117/volumes" Nov 02 15:36:09.592458 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:09.592426 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:11.015041 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:11.015001 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:36:12.858757 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:12.855659 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts"] Nov 02 15:36:13.162483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:13.162447 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:13.337646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:13.337608 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:15.377556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:15.377511 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:17.856565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:17.856528 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" podUID="e92810d1-ef00-4c5a-98cc-ea7eb2d7708a" containerName="payload-pre-processing" containerID="cri-o://0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5" gracePeriod=25 Nov 02 15:36:18.186673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.186635 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:36:18.191379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.191318 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-plugins-config-volume\") pod \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\" (UID: \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\") " Nov 02 15:36:18.191511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.191489 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-kube-api-access-6vrsv\" (UniqueName: \"kubernetes.io/projected/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-kube-api-access-6vrsv\") pod \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\" (UID: \"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a\") " Nov 02 15:36:18.191695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.191669 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-plugins-config-volume" pod "e92810d1-ef00-4c5a-98cc-ea7eb2d7708a" (UID: "e92810d1-ef00-4c5a-98cc-ea7eb2d7708a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:18.192028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.192007 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:18.193713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.193687 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-kube-api-access-6vrsv" pod "e92810d1-ef00-4c5a-98cc-ea7eb2d7708a" (UID: "e92810d1-ef00-4c5a-98cc-ea7eb2d7708a"). InnerVolumeSpecName "kube-api-access-6vrsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:18.293560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.293528 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vrsv\" (UniqueName: \"kubernetes.io/projected/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a-kube-api-access-6vrsv\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:18.873954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.873874 5433 generic.go:356] "Generic (PLEG): container finished" podID="e92810d1-ef00-4c5a-98cc-ea7eb2d7708a" containerID="0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5" exitCode=0 Nov 02 15:36:18.873954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.873926 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" Nov 02 15:36:18.873954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.873937 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" event={"ID":"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a","Type":"ContainerDied","Data":"0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5"} Nov 02 15:36:18.874544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.874011 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts" event={"ID":"e92810d1-ef00-4c5a-98cc-ea7eb2d7708a","Type":"ContainerDied","Data":"abfef5ee8f53f784032851210d00b5dd789a6b0d3932b647edc49c22ba383cd5"} Nov 02 15:36:18.874544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.874036 5433 scope.go:122] "RemoveContainer" containerID="0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5" Nov 02 15:36:18.884286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.884266 5433 scope.go:122] "RemoveContainer" containerID="0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5" Nov 02 15:36:18.884569 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:18.884547 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5\": container with ID starting with 0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5 not found: ID does not exist" containerID="0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5" Nov 02 15:36:18.884647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:18.884580 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5"} err="failed to get container status \"0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5\": rpc error: code = NotFound desc = could not find container \"0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5\": container with ID starting with 0d1fd58fc5445acf81052d6a1c7dd6183a37e0264c483ef493c08be8337e5fd5 not found: ID does not exist" Nov 02 15:36:19.045208 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:19.045176 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts"] Nov 02 15:36:19.089511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:19.089480 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-adf096-668ccbb4bc-tm7ts"] Nov 02 15:36:20.458678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:20.458642 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92810d1-ef00-4c5a-98cc-ea7eb2d7708a" path="/var/lib/kubelet/pods/e92810d1-ef00-4c5a-98cc-ea7eb2d7708a/volumes" Nov 02 15:36:20.481277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:20.481238 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:20.635159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:20.635112 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:20.797213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:20.797126 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:23.908025 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:23.907988 5433 generic.go:356] "Generic (PLEG): container finished" podID="232e57c9-f425-4638-89df-2eadc6a7821b" containerID="91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb" exitCode=0 Nov 02 15:36:23.908440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:23.908053 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" event={"ID":"232e57c9-f425-4638-89df-2eadc6a7821b","Type":"ContainerDied","Data":"91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb"} Nov 02 15:36:23.908484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:23.908452 5433 scope.go:122] "RemoveContainer" containerID="91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb" Nov 02 15:36:23.910007 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:23.909959 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:23.910122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:23.910066 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:23.910122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:23.910100 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:24.202743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.202668 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:36:24.395991 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.395948 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd"] Nov 02 15:36:24.396628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396606 5433 cpu_manager.go:414] "containerMap: removing container" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" Nov 02 15:36:24.396628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396625 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" Nov 02 15:36:24.396815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396637 5433 cpu_manager.go:414] "containerMap: removing container" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" Nov 02 15:36:24.396815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396644 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" Nov 02 15:36:24.396815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396664 5433 cpu_manager.go:414] "containerMap: removing container" podUID="e92810d1-ef00-4c5a-98cc-ea7eb2d7708a" containerName="payload-pre-processing" Nov 02 15:36:24.396815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396669 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92810d1-ef00-4c5a-98cc-ea7eb2d7708a" containerName="payload-pre-processing" Nov 02 15:36:24.396815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396783 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92810d1-ef00-4c5a-98cc-ea7eb2d7708a" containerName="payload-pre-processing" Nov 02 15:36:24.396815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396799 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="230df24c-62fc-42d9-a0a6-480d2e142269" containerName="istio-proxy" Nov 02 15:36:24.396815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396807 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" Nov 02 15:36:24.396815 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.396816 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="0516903c-1765-4aaf-9886-47e2ebfe0117" containerName="istio-proxy" Nov 02 15:36:24.400543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.400517 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.449916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.449889 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5aeac88e-openshift-default-dockercfg-hfkbc\"" Nov 02 15:36:24.459397 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.459327 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd"] Nov 02 15:36:24.479304 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.479273 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.479491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.479431 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.479491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.479471 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.479600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.479503 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.479646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.479596 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.479692 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.479643 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbhr\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-kube-api-access-bkbhr\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.479753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.479731 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.479843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.479825 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235eff14-e669-45ea-91db-8bd3ef7298d2-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.479919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.479902 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.581458 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581428 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.581644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581475 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.581644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581537 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.581644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581575 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.581644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581630 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbhr\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-kube-api-access-bkbhr\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.581863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581689 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.581863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581758 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235eff14-e669-45ea-91db-8bd3ef7298d2-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.581863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581814 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.582089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581877 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.582089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581903 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.582089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.581920 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.582402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.582380 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.582502 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.582477 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.582639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.582619 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235eff14-e669-45ea-91db-8bd3ef7298d2-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.583915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.583886 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.584144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.584128 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.638829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.638793 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.639030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.638992 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbhr\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-kube-api-access-bkbhr\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.715446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.715349 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:24.939999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:24.937366 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" event={"ID":"232e57c9-f425-4638-89df-2eadc6a7821b","Type":"ContainerStarted","Data":"14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d"} Nov 02 15:36:25.043867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.043764 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6"] Nov 02 15:36:25.044104 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.044053 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" podUID="e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" containerName="maas-api" containerID="cri-o://ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4" gracePeriod=30 Nov 02 15:36:25.190299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.190266 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:25.203597 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.203029 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:25.203597 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.203178 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:25.203597 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.203276 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:25.207724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.205034 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd"] Nov 02 15:36:25.420298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.419906 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:36:25.529042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.529005 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-metrics-certs\") pod \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " Nov 02 15:36:25.529794 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.529771 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-openshift-service-ca\") pod \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " Nov 02 15:36:25.530395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.530375 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls\") pod \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " Nov 02 15:36:25.530621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.530604 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-kube-api-access-2lfwb\" (UniqueName: \"kubernetes.io/projected/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-kube-api-access-2lfwb\") pod \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\" (UID: \"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4\") " Nov 02 15:36:25.531844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.530315 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-openshift-service-ca" pod "e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" (UID: "e2ef7d95-c50b-4f8c-8a89-1b8f573041d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:25.569680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.569608 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls" pod "e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" (UID: "e2ef7d95-c50b-4f8c-8a89-1b8f573041d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:36:25.570012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.569955 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-metrics-certs" pod "e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" (UID: "e2ef7d95-c50b-4f8c-8a89-1b8f573041d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:36:25.602057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.591365 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:36:25.606267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.605738 5433 patch_prober.go:29] interesting pod/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:36:25.606267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.605788 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 15:36:25.606267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.606194 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-kube-api-access-2lfwb" pod "e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" (UID: "e2ef7d95-c50b-4f8c-8a89-1b8f573041d4"). InnerVolumeSpecName "kube-api-access-2lfwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:25.665744 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.665689 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:25.665966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.665945 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:25.666130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.666114 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:25.666243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.666230 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lfwb\" (UniqueName: \"kubernetes.io/projected/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4-kube-api-access-2lfwb\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:25.949788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.949756 5433 generic.go:356] "Generic (PLEG): container finished" podID="e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" containerID="ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4" exitCode=0 Nov 02 15:36:25.950315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.949804 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" Nov 02 15:36:25.950315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.949847 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" event={"ID":"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4","Type":"ContainerDied","Data":"ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4"} Nov 02 15:36:25.950315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.949908 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6" event={"ID":"e2ef7d95-c50b-4f8c-8a89-1b8f573041d4","Type":"ContainerDied","Data":"de9a6a8ea58638de75d23c27014fcee51b59eb9cb7d298f2b0188bea598bdda9"} Nov 02 15:36:25.950315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.949939 5433 scope.go:122] "RemoveContainer" containerID="ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4" Nov 02 15:36:25.952024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.952004 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" event={"ID":"235eff14-e669-45ea-91db-8bd3ef7298d2","Type":"ContainerStarted","Data":"edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78"} Nov 02 15:36:25.952143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.952049 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" event={"ID":"235eff14-e669-45ea-91db-8bd3ef7298d2","Type":"ContainerStarted","Data":"795854b853c561f3fa06a966ace8519567c361ed80e8634a122f91c6a673dcc4"} Nov 02 15:36:25.960469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.960451 5433 scope.go:122] "RemoveContainer" containerID="ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4" Nov 02 15:36:25.960794 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:25.960775 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4\": container with ID starting with ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4 not found: ID does not exist" containerID="ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4" Nov 02 15:36:25.960865 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:25.960810 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4"} err="failed to get container status \"ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4\": rpc error: code = NotFound desc = could not find container \"ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4\": container with ID starting with ede4eb3152c0a5770a565d575009cc5d91b966974f2dcd2b18900e72188ad4c4 not found: ID does not exist" Nov 02 15:36:26.129698 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.129631 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podStartSLOduration=2.12960468 podStartE2EDuration="2.12960468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:26.128483472 +0000 UTC m=+2289.801686670" watchObservedRunningTime="2026-11-02 15:36:26.12960468 +0000 UTC m=+2289.802807940" Nov 02 15:36:26.254255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.254155 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6"] Nov 02 15:36:26.257245 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.257212 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1c449c-759966475b-g5ds6"] Nov 02 15:36:26.458248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.458215 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" path="/var/lib/kubelet/pods/e2ef7d95-c50b-4f8c-8a89-1b8f573041d4/volumes" Nov 02 15:36:26.583625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.583548 5433 patch_prober.go:29] interesting pod/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:36:26.583625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.583600 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 15:36:26.715727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.715696 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:26.738316 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.738279 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 15:36:26.742223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.738579 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 15:36:26.835058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:26.834734 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:27.054684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:27.054647 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:27.219485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:27.219449 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:27.583189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:27.583105 5433 patch_prober.go:29] interesting pod/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:36:27.583189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:27.583159 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 15:36:27.715831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:27.715800 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 15:36:27.716032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:27.715859 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 15:36:27.773556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:27.773520 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:28.583582 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:28.583537 5433 patch_prober.go:29] interesting pod/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:36:28.584133 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:28.583600 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 15:36:28.715915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:28.715885 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 15:36:28.737138 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:28.737088 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 15:36:29.115053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:29.114967 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:29.583655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:29.583618 5433 patch_prober.go:29] interesting pod/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:36:29.584196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:29.583684 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 15:36:29.743344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:29.743298 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 15:36:29.743840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:29.743803 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 15:36:30.588162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:30.588131 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:36:30.588667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:30.588534 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:36:30.589121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:30.589098 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:36:30.741474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:30.741439 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 15:36:30.741625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:30.741500 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 15:36:31.736999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:31.736094 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 15:36:31.736999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:31.736156 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 15:36:32.716051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:32.716005 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 15:36:32.716234 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:32.716078 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 15:36:32.814893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:32.814853 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:33.721552 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:33.721522 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:33.722024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:33.721970 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:33.722746 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:33.722718 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:36:34.028463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:34.028366 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:34.197500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:34.197457 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:34.337779 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:34.337685 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v"] Nov 02 15:36:34.337959 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:34.337903 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" containerID="cri-o://c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6" gracePeriod=30 Nov 02 15:36:34.356412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:34.356384 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:34.721639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:34.721598 5433 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:36:34.721822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:34.721674 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:36:37.505192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:37.505130 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:38.303109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:38.303074 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:39.529925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.528460 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w"] Nov 02 15:36:39.529925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.529726 5433 cpu_manager.go:414] "containerMap: removing container" podUID="e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" containerName="maas-api" Nov 02 15:36:39.529925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.529749 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" containerName="maas-api" Nov 02 15:36:39.529925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.529880 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ef7d95-c50b-4f8c-8a89-1b8f573041d4" containerName="maas-api" Nov 02 15:36:39.534089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.534062 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.588417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.588383 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w"] Nov 02 15:36:39.637906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.637874 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.638080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.638004 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8066dd3-914f-4f57-aa28-b5df71b7380d-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.638164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.638080 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.638227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.638171 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.638290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.638234 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.638355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.638336 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77647\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-kube-api-access-77647\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.638413 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.638372 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.638464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.638438 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.638524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.638485 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.646514 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.646490 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:36:39.740746 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.740682 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-token\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-token\") pod \"04a39989-2f95-40cb-a48d-98addc6d6f34\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " Nov 02 15:36:39.740918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.740795 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-credential-socket\") pod \"04a39989-2f95-40cb-a48d-98addc6d6f34\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " Nov 02 15:36:39.740918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.740888 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-socket\") pod \"04a39989-2f95-40cb-a48d-98addc6d6f34\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " Nov 02 15:36:39.741076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.740954 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-certs\") pod \"04a39989-2f95-40cb-a48d-98addc6d6f34\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " Nov 02 15:36:39.741076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741047 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-podinfo\") pod \"04a39989-2f95-40cb-a48d-98addc6d6f34\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " Nov 02 15:36:39.741201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741122 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a39989-2f95-40cb-a48d-98addc6d6f34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a39989-2f95-40cb-a48d-98addc6d6f34-istiod-ca-cert\") pod \"04a39989-2f95-40cb-a48d-98addc6d6f34\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " Nov 02 15:36:39.741201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741138 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-credential-socket" pod "04a39989-2f95-40cb-a48d-98addc6d6f34" (UID: "04a39989-2f95-40cb-a48d-98addc6d6f34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:39.741295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741187 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-socket" pod "04a39989-2f95-40cb-a48d-98addc6d6f34" (UID: "04a39989-2f95-40cb-a48d-98addc6d6f34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:39.741295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741244 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-envoy\") pod \"04a39989-2f95-40cb-a48d-98addc6d6f34\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " Nov 02 15:36:39.741295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741250 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-certs" pod "04a39989-2f95-40cb-a48d-98addc6d6f34" (UID: "04a39989-2f95-40cb-a48d-98addc6d6f34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:39.741465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741367 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-data\") pod \"04a39989-2f95-40cb-a48d-98addc6d6f34\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " Nov 02 15:36:39.741465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741400 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-kube-api-access-p5n9p\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-kube-api-access-p5n9p\") pod \"04a39989-2f95-40cb-a48d-98addc6d6f34\" (UID: \"04a39989-2f95-40cb-a48d-98addc6d6f34\") " Nov 02 15:36:39.741564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741503 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a39989-2f95-40cb-a48d-98addc6d6f34-istiod-ca-cert" pod "04a39989-2f95-40cb-a48d-98addc6d6f34" (UID: "04a39989-2f95-40cb-a48d-98addc6d6f34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:39.741665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741645 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.741790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741707 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.741855 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741806 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.741912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741871 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8066dd3-914f-4f57-aa28-b5df71b7380d-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.741912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741878 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-data" pod "04a39989-2f95-40cb-a48d-98addc6d6f34" (UID: "04a39989-2f95-40cb-a48d-98addc6d6f34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:39.742075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.741955 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.742075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742036 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.742184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742090 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.742184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742126 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.742184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742144 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.742354 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742214 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77647\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-kube-api-access-77647\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.742354 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742264 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.742456 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742411 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.742456 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742430 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a39989-2f95-40cb-a48d-98addc6d6f34-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.742456 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742446 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.742607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742463 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.742607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742478 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.742607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742572 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8066dd3-914f-4f57-aa28-b5df71b7380d-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.742846 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.742822 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.743163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.743139 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.743677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.743650 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-token" pod "04a39989-2f95-40cb-a48d-98addc6d6f34" (UID: "04a39989-2f95-40cb-a48d-98addc6d6f34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:39.743789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.743764 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-envoy" pod "04a39989-2f95-40cb-a48d-98addc6d6f34" (UID: "04a39989-2f95-40cb-a48d-98addc6d6f34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:36:39.743849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.743826 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-podinfo" pod "04a39989-2f95-40cb-a48d-98addc6d6f34" (UID: "04a39989-2f95-40cb-a48d-98addc6d6f34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:36:39.744375 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.744350 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-kube-api-access-p5n9p" pod "04a39989-2f95-40cb-a48d-98addc6d6f34" (UID: "04a39989-2f95-40cb-a48d-98addc6d6f34"). InnerVolumeSpecName "kube-api-access-p5n9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:39.744478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.744402 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.745309 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.745288 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.799698 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.799671 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.800214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.800191 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77647\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-kube-api-access-77647\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:39.843757 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.843722 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.843757 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.843755 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.843757 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.843765 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5n9p\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-kube-api-access-p5n9p\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.844032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.843775 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a39989-2f95-40cb-a48d-98addc6d6f34-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.866351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:39.866269 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:40.062092 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.062057 5433 generic.go:356] "Generic (PLEG): container finished" podID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerID="c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6" exitCode=0 Nov 02 15:36:40.062268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.062126 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" Nov 02 15:36:40.062268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.062157 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" event={"ID":"04a39989-2f95-40cb-a48d-98addc6d6f34","Type":"ContainerDied","Data":"c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6"} Nov 02 15:36:40.062268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.062216 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v" event={"ID":"04a39989-2f95-40cb-a48d-98addc6d6f34","Type":"ContainerDied","Data":"613ba7974c16515f4cd68a7994a54ee0416af5e0341ac3f15f10c98a38937022"} Nov 02 15:36:40.062268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.062239 5433 scope.go:122] "RemoveContainer" containerID="c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6" Nov 02 15:36:40.073576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.073559 5433 scope.go:122] "RemoveContainer" containerID="3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7" Nov 02 15:36:40.083045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.083024 5433 scope.go:122] "RemoveContainer" containerID="c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6" Nov 02 15:36:40.083335 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:40.083310 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6\": container with ID starting with c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6 not found: ID does not exist" containerID="c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6" Nov 02 15:36:40.083387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.083346 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6"} err="failed to get container status \"c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6\": rpc error: code = NotFound desc = could not find container \"c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6\": container with ID starting with c8129469e0c4ca2175a0970d40fe791436eb91e082a554e99178b2267b32ecd6 not found: ID does not exist" Nov 02 15:36:40.083387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.083367 5433 scope.go:122] "RemoveContainer" containerID="3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7" Nov 02 15:36:40.083604 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:40.083586 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7\": container with ID starting with 3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7 not found: ID does not exist" containerID="3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7" Nov 02 15:36:40.083647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.083613 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7"} err="failed to get container status \"3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7\": rpc error: code = NotFound desc = could not find container \"3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7\": container with ID starting with 3489499d097b13ff11fb69136e67ea781878652c163b6c3e2e02ac7df3c00ff7 not found: ID does not exist" Nov 02 15:36:40.218241 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.218207 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v"] Nov 02 15:36:40.263586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.263543 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1c449c-openshift-default-8cc6f8dcf-7c57v"] Nov 02 15:36:40.286553 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.286522 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w"] Nov 02 15:36:40.288196 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:36:40.288168 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8066dd3_914f_4f57_aa28_b5df71b7380d.slice/crio-e3e9f957c6b56feecd8ec376b407e39285013a8eef01fd7e8abb70110d97d4d2 WatchSource:0}: Error finding container e3e9f957c6b56feecd8ec376b407e39285013a8eef01fd7e8abb70110d97d4d2: Status 404 returned error can't find the container with id e3e9f957c6b56feecd8ec376b407e39285013a8eef01fd7e8abb70110d97d4d2 Nov 02 15:36:40.291082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.291043 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:40.291187 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.291112 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:40.291187 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.291142 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:40.457886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:40.457854 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" path="/var/lib/kubelet/pods/04a39989-2f95-40cb-a48d-98addc6d6f34/volumes" Nov 02 15:36:41.013129 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:41.013096 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:36:41.070897 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:41.070858 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" event={"ID":"e8066dd3-914f-4f57-aa28-b5df71b7380d","Type":"ContainerStarted","Data":"460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2"} Nov 02 15:36:41.071108 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:41.070926 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" event={"ID":"e8066dd3-914f-4f57-aa28-b5df71b7380d","Type":"ContainerStarted","Data":"e3e9f957c6b56feecd8ec376b407e39285013a8eef01fd7e8abb70110d97d4d2"} Nov 02 15:36:41.207222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:41.207164 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" podStartSLOduration=2.207146043 podStartE2EDuration="2.207146043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:41.205577819 +0000 UTC m=+2304.878781027" watchObservedRunningTime="2026-11-02 15:36:41.207146043 +0000 UTC m=+2304.880349244" Nov 02 15:36:41.867061 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:41.867021 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:41.868806 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:41.868780 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 15:36:41.868882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:41.868839 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 15:36:42.866823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:42.866783 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 15:36:42.867377 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:42.866845 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 15:36:43.288221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.288177 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24"] Nov 02 15:36:43.288947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.288923 5433 cpu_manager.go:414] "containerMap: removing container" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" Nov 02 15:36:43.289082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.288948 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" Nov 02 15:36:43.289082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.288998 5433 cpu_manager.go:414] "containerMap: removing container" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" Nov 02 15:36:43.289082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.289007 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" Nov 02 15:36:43.289241 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.289172 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" Nov 02 15:36:43.289241 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.289191 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a39989-2f95-40cb-a48d-98addc6d6f34" containerName="istio-proxy" Nov 02 15:36:43.293157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.293135 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.342022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.341954 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2203c8-serving-cert\"" Nov 02 15:36:43.392348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.392290 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24"] Nov 02 15:36:43.392348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.392319 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07fa338b-7a5f-42de-b42b-8037d94048d9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.392586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.392402 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-metrics-certs\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.392586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.392471 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67mq\" (UniqueName: \"kubernetes.io/projected/07fa338b-7a5f-42de-b42b-8037d94048d9-kube-api-access-s67mq\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.392751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.392735 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.494408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.494374 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.494585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.494450 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07fa338b-7a5f-42de-b42b-8037d94048d9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.494585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.494468 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-metrics-certs\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.494585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.494498 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67mq\" (UniqueName: \"kubernetes.io/projected/07fa338b-7a5f-42de-b42b-8037d94048d9-kube-api-access-s67mq\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.494585 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:43.494518 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-serving-cert: secret "maas-api-e2e-inf-a-2203c8-serving-cert" not found Nov 02 15:36:43.494585 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:36:43.494586 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls podName:07fa338b-7a5f-42de-b42b-8037d94048d9 nodeName:}" failed. No retries permitted until 2026-11-02 15:36:43.994565871 +0000 UTC m=+2307.667769044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls") pod "maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" (UID: "07fa338b-7a5f-42de-b42b-8037d94048d9") : secret "maas-api-e2e-inf-a-2203c8-serving-cert" not found Nov 02 15:36:43.495241 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.495221 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07fa338b-7a5f-42de-b42b-8037d94048d9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.496912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.496891 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-metrics-certs\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.498048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.498027 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:43.583564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.582754 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67mq\" (UniqueName: \"kubernetes.io/projected/07fa338b-7a5f-42de-b42b-8037d94048d9-kube-api-access-s67mq\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:43.680510 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.680476 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:43.842704 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.842602 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:43.867631 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.867597 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 15:36:43.868068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:43.867661 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 15:36:44.003487 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:44.003449 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:44.005893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:44.005862 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls\") pod \"maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:44.209112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:44.209075 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:44.608464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:44.608433 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24"] Nov 02 15:36:44.610181 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:36:44.610148 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fa338b_7a5f_42de_b42b_8037d94048d9.slice/crio-f76838d56cb7e59fc48c463aa97f216b999303b4e2a5d307454fffed3092779f WatchSource:0}: Error finding container f76838d56cb7e59fc48c463aa97f216b999303b4e2a5d307454fffed3092779f: Status 404 returned error can't find the container with id f76838d56cb7e59fc48c463aa97f216b999303b4e2a5d307454fffed3092779f Nov 02 15:36:44.867432 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:44.867353 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 15:36:44.867432 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:44.867406 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 15:36:45.095956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:45.095923 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" event={"ID":"07fa338b-7a5f-42de-b42b-8037d94048d9","Type":"ContainerStarted","Data":"f76838d56cb7e59fc48c463aa97f216b999303b4e2a5d307454fffed3092779f"} Nov 02 15:36:45.866819 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:45.866737 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 15:36:45.866819 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:45.866792 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 15:36:46.105130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:46.105094 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" event={"ID":"07fa338b-7a5f-42de-b42b-8037d94048d9","Type":"ContainerStarted","Data":"45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238"} Nov 02 15:36:46.105566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:46.105230 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:46.235155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:46.230755 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" podStartSLOduration=2.274917561 podStartE2EDuration="3.230731971s" totalImagesPullingTime="955.81441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:46.229268861 +0000 UTC m=+2309.902472058" watchObservedRunningTime="2026-11-02 15:36:46.230731971 +0000 UTC m=+2309.903935168" Nov 02 15:36:46.873063 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:46.873034 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:46.873454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:46.873436 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:46.874026 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:46.874007 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:36:47.921199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:47.921168 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:48.076834 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:48.076799 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:48.248708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:48.248627 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:49.885992 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:49.885949 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:50.177411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:50.177297 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:50.317479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:50.317433 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:50.825891 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:50.825860 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:52.118359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:52.118332 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:36:54.640594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.640564 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp"] Nov 02 15:36:54.645161 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.645141 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.741615 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.741577 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.741832 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.741641 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.741832 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.741742 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.741832 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.741823 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.742065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.741887 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.742065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.742032 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v59xj\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-kube-api-access-v59xj\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.742181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.742097 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.742181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.742163 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.742286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.742247 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef077e5-f94e-4e86-8d38-69fdda900991-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.744793 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.744771 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp"] Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.850340 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.850420 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.850457 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.850509 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.850577 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.850646 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59xj\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-kube-api-access-v59xj\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.850703 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.850830 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.850855 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef077e5-f94e-4e86-8d38-69fdda900991-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.851808 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef077e5-f94e-4e86-8d38-69fdda900991-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.852413 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.852685 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.872001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.852905 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.884994 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.880619 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.916040 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.911690 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.929470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.929433 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.969689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.969608 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:54.982624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:54.970949 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59xj\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-kube-api-access-v59xj\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:55.258829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:55.258727 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:55.729651 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:36:55.729610 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef077e5_f94e_4e86_8d38_69fdda900991.slice/crio-58730d99fc64ca7101fa5d00837389eb0872db2e3f643e33b621f8b98397c82c WatchSource:0}: Error finding container 58730d99fc64ca7101fa5d00837389eb0872db2e3f643e33b621f8b98397c82c: Status 404 returned error can't find the container with id 58730d99fc64ca7101fa5d00837389eb0872db2e3f643e33b621f8b98397c82c Nov 02 15:36:55.732595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:55.732561 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:55.732696 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:55.732634 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:55.732696 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:55.732664 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:55.770488 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:55.770458 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp"] Nov 02 15:36:55.881333 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:55.881274 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:56.226270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:56.226232 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" event={"ID":"5ef077e5-f94e-4e86-8d38-69fdda900991","Type":"ContainerStarted","Data":"4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25"} Nov 02 15:36:56.226449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:56.226294 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" event={"ID":"5ef077e5-f94e-4e86-8d38-69fdda900991","Type":"ContainerStarted","Data":"58730d99fc64ca7101fa5d00837389eb0872db2e3f643e33b621f8b98397c82c"} Nov 02 15:36:56.259805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:56.259773 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:36:56.261306 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:56.261278 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 15:36:56.261438 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:56.261335 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 15:36:56.376408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:56.376345 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podStartSLOduration=2.376324639 podStartE2EDuration="2.376324639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:56.372620419 +0000 UTC m=+2320.045823615" watchObservedRunningTime="2026-11-02 15:36:56.376324639 +0000 UTC m=+2320.049527837" Nov 02 15:36:57.259585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:57.259554 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 15:36:57.259956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:57.259603 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 15:36:57.401286 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:57.401249 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:57.542649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:57.542557 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:57.747310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:57.747267 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:58.259673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:58.259639 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 15:36:58.260110 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:58.259688 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 15:36:58.603674 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:58.603587 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:59.154025 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.150539 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7"] Nov 02 15:36:59.183298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.183265 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.355135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.336238 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-envoy\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.355135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.336344 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-data\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.355135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.336437 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-token\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.355135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.336467 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77r8p\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-kube-api-access-77r8p\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.355135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.336543 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-credential-socket\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.355135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.336632 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-certs\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.355135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.336658 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-podinfo\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.355135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.336712 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-socket\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.355135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.336788 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istiod-ca-cert\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.404375 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.404287 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:59.408310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.408123 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 15:36:59.408310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.408168 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 15:36:59.416852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.416821 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2c35ee-openshift-default-dockercfg-dlmrw\"" Nov 02 15:36:59.434037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.432039 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7"] Nov 02 15:36:59.589054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.514042 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-envoy\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.589262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.589239 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-data\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.589425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.589390 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-token\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.589507 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.589429 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77r8p\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-kube-api-access-77r8p\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.589567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.589551 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-credential-socket\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.589628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.589615 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-certs\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.589691 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.589639 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-podinfo\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.589748 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.589700 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-socket\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.589828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.589797 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istiod-ca-cert\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.590633 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.590600 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istiod-ca-cert\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.590938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.590913 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-certs\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.591614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.591078 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:36:59.591614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.591409 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-credential-socket\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.591614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.591556 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-data\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.591835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.591738 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-socket\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.592515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.592493 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-envoy\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.595907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.595822 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-podinfo\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.650412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.650374 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-token\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.650576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.650457 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77r8p\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-kube-api-access-77r8p\") pod \"e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:36:59.890445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:36:59.890413 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:37:00.259545 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.259466 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 15:37:00.259545 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.259522 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 15:37:00.345904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.345872 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7"] Nov 02 15:37:00.347901 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:37:00.347872 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1aacac_a3f6_4b64_84ed_2c673e7edd59.slice/crio-5027bf15518a36616c9c2b8f8b891625635efc956b4cd405aa6c3e2de2df1658 WatchSource:0}: Error finding container 5027bf15518a36616c9c2b8f8b891625635efc956b4cd405aa6c3e2de2df1658: Status 404 returned error can't find the container with id 5027bf15518a36616c9c2b8f8b891625635efc956b4cd405aa6c3e2de2df1658 Nov 02 15:37:00.350667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.350637 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:37:00.350741 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.350707 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:37:00.350780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.350738 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:37:00.460011 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.459952 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" event={"ID":"4c1aacac-a3f6-4b64-84ed-2c673e7edd59","Type":"ContainerStarted","Data":"978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c"} Nov 02 15:37:00.460434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.460035 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" event={"ID":"4c1aacac-a3f6-4b64-84ed-2c673e7edd59","Type":"ContainerStarted","Data":"5027bf15518a36616c9c2b8f8b891625635efc956b4cd405aa6c3e2de2df1658"} Nov 02 15:37:00.577109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.577017 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" podStartSLOduration=1.5770015160000002 podStartE2EDuration="1.577001516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:00.575126024 +0000 UTC m=+2324.248329220" watchObservedRunningTime="2026-11-02 15:37:00.577001516 +0000 UTC m=+2324.250204713" Nov 02 15:37:00.891369 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:00.891337 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:37:01.259879 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:01.259758 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 15:37:01.259879 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:01.259824 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 15:37:01.666384 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:01.666336 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:01.897386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:01.897343 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:37:02.260062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:02.260019 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 15:37:02.260236 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:02.260076 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 15:37:02.482894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:02.482855 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:37:02.485190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:02.485156 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:37:03.264229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:03.264183 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:37:03.264721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:03.264569 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:37:03.265317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:03.265288 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:37:03.437449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:03.437410 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:03.623379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:03.623282 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:03.800022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:03.794180 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:06.665279 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:06.665238 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:06.899673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:06.899380 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:07.326237 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.326202 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8"] Nov 02 15:37:07.350499 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.347129 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8"] Nov 02 15:37:07.350499 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.347251 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:07.398714 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.398677 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2c35ee\"" Nov 02 15:37:07.444817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.444786 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2c35ee-dockercfg-4zkkx\"" Nov 02 15:37:07.508112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.508072 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54066c7-e857-4a05-a72c-f194d6a6b655-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8\" (UID: \"b54066c7-e857-4a05-a72c-f194d6a6b655\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:07.508317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.508242 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr24z\" (UniqueName: \"kubernetes.io/projected/b54066c7-e857-4a05-a72c-f194d6a6b655-kube-api-access-gr24z\") pod \"payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8\" (UID: \"b54066c7-e857-4a05-a72c-f194d6a6b655\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:07.609833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.609732 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54066c7-e857-4a05-a72c-f194d6a6b655-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8\" (UID: \"b54066c7-e857-4a05-a72c-f194d6a6b655\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:07.610050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.609856 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr24z\" (UniqueName: \"kubernetes.io/projected/b54066c7-e857-4a05-a72c-f194d6a6b655-kube-api-access-gr24z\") pod \"payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8\" (UID: \"b54066c7-e857-4a05-a72c-f194d6a6b655\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:07.610509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.610489 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54066c7-e857-4a05-a72c-f194d6a6b655-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8\" (UID: \"b54066c7-e857-4a05-a72c-f194d6a6b655\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:07.667877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.667838 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr24z\" (UniqueName: \"kubernetes.io/projected/b54066c7-e857-4a05-a72c-f194d6a6b655-kube-api-access-gr24z\") pod \"payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8\" (UID: \"b54066c7-e857-4a05-a72c-f194d6a6b655\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:07.680055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.680026 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:07.683640 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:07.683270 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:08.080174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:08.080138 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8"] Nov 02 15:37:08.080617 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:37:08.080579 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54066c7_e857_4a05_a72c_f194d6a6b655.slice/crio-ba8bee49556e6b3bff7cc3e877e1b925973d36de7aec2299a9019a1c7a569e5a WatchSource:0}: Error finding container ba8bee49556e6b3bff7cc3e877e1b925973d36de7aec2299a9019a1c7a569e5a: Status 404 returned error can't find the container with id ba8bee49556e6b3bff7cc3e877e1b925973d36de7aec2299a9019a1c7a569e5a Nov 02 15:37:08.524460 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:08.524422 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" event={"ID":"b54066c7-e857-4a05-a72c-f194d6a6b655","Type":"ContainerStarted","Data":"2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134"} Nov 02 15:37:08.524673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:08.524471 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" event={"ID":"b54066c7-e857-4a05-a72c-f194d6a6b655","Type":"ContainerStarted","Data":"ba8bee49556e6b3bff7cc3e877e1b925973d36de7aec2299a9019a1c7a569e5a"} Nov 02 15:37:08.524673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:08.524500 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:08.635628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:08.635572 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" podStartSLOduration=1.635557527 podStartE2EDuration="1.635557527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:08.632911033 +0000 UTC m=+2332.306114228" watchObservedRunningTime="2026-11-02 15:37:08.635557527 +0000 UTC m=+2332.308760721" Nov 02 15:37:11.012533 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:37:11.012487 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:37:12.721849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:12.721811 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t"] Nov 02 15:37:12.829124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:12.829092 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc"] Nov 02 15:37:12.868543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:12.868503 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:13.109226 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:13.109130 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:13.262003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:13.250680 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:17.722233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:17.722187 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" podUID="82934be9-75b5-4023-8013-2ae3c5456011" containerName="payload-processing" containerID="cri-o://3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2" gracePeriod=25 Nov 02 15:37:17.829682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:17.829599 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" podUID="7ba9a13a-5bd4-4d29-aa48-b10162823acf" containerName="payload-pre-processing" containerID="cri-o://d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b" gracePeriod=25 Nov 02 15:37:18.035937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.035904 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:37:18.041702 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.041664 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:18.170838 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.166832 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82934be9-75b5-4023-8013-2ae3c5456011-kube-api-access-9gkhb\" (UniqueName: \"kubernetes.io/projected/82934be9-75b5-4023-8013-2ae3c5456011-kube-api-access-9gkhb\") pod \"82934be9-75b5-4023-8013-2ae3c5456011\" (UID: \"82934be9-75b5-4023-8013-2ae3c5456011\") " Nov 02 15:37:18.175298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.175268 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82934be9-75b5-4023-8013-2ae3c5456011-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82934be9-75b5-4023-8013-2ae3c5456011-plugins-config-volume\") pod \"82934be9-75b5-4023-8013-2ae3c5456011\" (UID: \"82934be9-75b5-4023-8013-2ae3c5456011\") " Nov 02 15:37:18.192598 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.179809 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82934be9-75b5-4023-8013-2ae3c5456011-plugins-config-volume" pod "82934be9-75b5-4023-8013-2ae3c5456011" (UID: "82934be9-75b5-4023-8013-2ae3c5456011"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:18.192598 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.187520 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82934be9-75b5-4023-8013-2ae3c5456011-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:18.205894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.205873 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:37:18.210693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.210645 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82934be9-75b5-4023-8013-2ae3c5456011-kube-api-access-9gkhb" pod "82934be9-75b5-4023-8013-2ae3c5456011" (UID: "82934be9-75b5-4023-8013-2ae3c5456011"). InnerVolumeSpecName "kube-api-access-9gkhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:18.288765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.288686 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba9a13a-5bd4-4d29-aa48-b10162823acf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba9a13a-5bd4-4d29-aa48-b10162823acf-plugins-config-volume\") pod \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\" (UID: \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\") " Nov 02 15:37:18.288944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.288762 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba9a13a-5bd4-4d29-aa48-b10162823acf-kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/7ba9a13a-5bd4-4d29-aa48-b10162823acf-kube-api-access-x9mq6\") pod \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\" (UID: \"7ba9a13a-5bd4-4d29-aa48-b10162823acf\") " Nov 02 15:37:18.289160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.289137 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba9a13a-5bd4-4d29-aa48-b10162823acf-plugins-config-volume" pod "7ba9a13a-5bd4-4d29-aa48-b10162823acf" (UID: "7ba9a13a-5bd4-4d29-aa48-b10162823acf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:18.289520 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.289498 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gkhb\" (UniqueName: \"kubernetes.io/projected/82934be9-75b5-4023-8013-2ae3c5456011-kube-api-access-9gkhb\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:18.289583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.289526 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba9a13a-5bd4-4d29-aa48-b10162823acf-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:18.290909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.290889 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba9a13a-5bd4-4d29-aa48-b10162823acf-kube-api-access-x9mq6" pod "7ba9a13a-5bd4-4d29-aa48-b10162823acf" (UID: "7ba9a13a-5bd4-4d29-aa48-b10162823acf"). InnerVolumeSpecName "kube-api-access-x9mq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:18.390990 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.390939 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9mq6\" (UniqueName: \"kubernetes.io/projected/7ba9a13a-5bd4-4d29-aa48-b10162823acf-kube-api-access-x9mq6\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:18.620248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.620159 5433 generic.go:356] "Generic (PLEG): container finished" podID="82934be9-75b5-4023-8013-2ae3c5456011" containerID="3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2" exitCode=0 Nov 02 15:37:18.620248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.620229 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" Nov 02 15:37:18.620248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.620237 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" event={"ID":"82934be9-75b5-4023-8013-2ae3c5456011","Type":"ContainerDied","Data":"3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2"} Nov 02 15:37:18.620522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.620297 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t" event={"ID":"82934be9-75b5-4023-8013-2ae3c5456011","Type":"ContainerDied","Data":"e96d76fe3f71916a9dd368d6814cc52bf53592adcbb5c8253dc199ab955f9bcb"} Nov 02 15:37:18.620522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.620329 5433 scope.go:122] "RemoveContainer" containerID="3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2" Nov 02 15:37:18.621778 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.621756 5433 generic.go:356] "Generic (PLEG): container finished" podID="7ba9a13a-5bd4-4d29-aa48-b10162823acf" containerID="d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b" exitCode=0 Nov 02 15:37:18.621906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.621808 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" Nov 02 15:37:18.621906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.621849 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" event={"ID":"7ba9a13a-5bd4-4d29-aa48-b10162823acf","Type":"ContainerDied","Data":"d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b"} Nov 02 15:37:18.621906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.621892 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" event={"ID":"7ba9a13a-5bd4-4d29-aa48-b10162823acf","Type":"ContainerDied","Data":"7d0b9b5ac55c77b747e21aa5fd5a28ace8f9416f736d991cb01ed9e7c30a6d03"} Nov 02 15:37:18.630292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.630274 5433 scope.go:122] "RemoveContainer" containerID="3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2" Nov 02 15:37:18.630576 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:37:18.630556 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2\": container with ID starting with 3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2 not found: ID does not exist" containerID="3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2" Nov 02 15:37:18.630634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.630586 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2"} err="failed to get container status \"3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2\": rpc error: code = NotFound desc = could not find container \"3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2\": container with ID starting with 3865dd43edf23f468a7cf1c1620e00592d92e9a9016bdc85dabc185a877874a2 not found: ID does not exist" Nov 02 15:37:18.630634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.630603 5433 scope.go:122] "RemoveContainer" containerID="d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b" Nov 02 15:37:18.639450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.639433 5433 scope.go:122] "RemoveContainer" containerID="d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b" Nov 02 15:37:18.639744 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:37:18.639719 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b\": container with ID starting with d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b not found: ID does not exist" containerID="d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b" Nov 02 15:37:18.639828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.639749 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b"} err="failed to get container status \"d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b\": rpc error: code = NotFound desc = could not find container \"d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b\": container with ID starting with d67f6c6cef695f621c36b8f42d6058719e3e29399c98e143c05c2e9dda1dce8b not found: ID does not exist" Nov 02 15:37:18.685778 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.685746 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk"] Nov 02 15:37:18.775018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.774986 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t"] Nov 02 15:37:18.778849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.778824 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-qlr8t"] Nov 02 15:37:18.883420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.883383 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc"] Nov 02 15:37:18.888229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.888197 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc"] Nov 02 15:37:18.930592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:18.930558 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:19.074799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:19.074767 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:19.196082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:19.195954 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5aeac88e-5c495779-5kmkc" podUID="7ba9a13a-5bd4-4d29-aa48-b10162823acf" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.123:50052\" within 1s: context deadline exceeded" Nov 02 15:37:19.196082 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:37:19.196036 5433 logging.go:55] [core] [Channel #4220 SubChannel #4221]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.123:50052", ServerName: "10.133.0.123:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.123:50052: operation was canceled" Nov 02 15:37:19.535859 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:19.535775 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:37:20.464743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:20.464408 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba9a13a-5bd4-4d29-aa48-b10162823acf" path="/var/lib/kubelet/pods/7ba9a13a-5bd4-4d29-aa48-b10162823acf/volumes" Nov 02 15:37:20.465512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:20.464942 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82934be9-75b5-4023-8013-2ae3c5456011" path="/var/lib/kubelet/pods/82934be9-75b5-4023-8013-2ae3c5456011/volumes" Nov 02 15:37:23.092215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:23.092175 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:23.686650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:23.686596 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" podUID="4b887d84-52a2-415a-99c7-4bb5de296ba3" containerName="payload-pre-processing" containerID="cri-o://f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd" gracePeriod=25 Nov 02 15:37:23.961096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:23.961068 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:37:24.110594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.110558 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b887d84-52a2-415a-99c7-4bb5de296ba3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b887d84-52a2-415a-99c7-4bb5de296ba3-plugins-config-volume\") pod \"4b887d84-52a2-415a-99c7-4bb5de296ba3\" (UID: \"4b887d84-52a2-415a-99c7-4bb5de296ba3\") " Nov 02 15:37:24.111029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.110618 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b887d84-52a2-415a-99c7-4bb5de296ba3-kube-api-access-f2rhl\" (UniqueName: \"kubernetes.io/projected/4b887d84-52a2-415a-99c7-4bb5de296ba3-kube-api-access-f2rhl\") pod \"4b887d84-52a2-415a-99c7-4bb5de296ba3\" (UID: \"4b887d84-52a2-415a-99c7-4bb5de296ba3\") " Nov 02 15:37:24.111029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.110907 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b887d84-52a2-415a-99c7-4bb5de296ba3-plugins-config-volume" pod "4b887d84-52a2-415a-99c7-4bb5de296ba3" (UID: "4b887d84-52a2-415a-99c7-4bb5de296ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:24.111239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.111221 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b887d84-52a2-415a-99c7-4bb5de296ba3-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:24.112893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.112863 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b887d84-52a2-415a-99c7-4bb5de296ba3-kube-api-access-f2rhl" pod "4b887d84-52a2-415a-99c7-4bb5de296ba3" (UID: "4b887d84-52a2-415a-99c7-4bb5de296ba3"). InnerVolumeSpecName "kube-api-access-f2rhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:24.212342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.212249 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2rhl\" (UniqueName: \"kubernetes.io/projected/4b887d84-52a2-415a-99c7-4bb5de296ba3-kube-api-access-f2rhl\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:24.677333 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.677293 5433 generic.go:356] "Generic (PLEG): container finished" podID="4b887d84-52a2-415a-99c7-4bb5de296ba3" containerID="f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd" exitCode=0 Nov 02 15:37:24.677496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.677341 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" Nov 02 15:37:24.677496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.677374 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" event={"ID":"4b887d84-52a2-415a-99c7-4bb5de296ba3","Type":"ContainerDied","Data":"f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd"} Nov 02 15:37:24.677496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.677434 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk" event={"ID":"4b887d84-52a2-415a-99c7-4bb5de296ba3","Type":"ContainerDied","Data":"bd6bc06b1d508022e00d2cb7d1e305959d0ef102a333079bcc67ecce15f5df83"} Nov 02 15:37:24.677496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.677456 5433 scope.go:122] "RemoveContainer" containerID="f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd" Nov 02 15:37:24.687290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.687095 5433 scope.go:122] "RemoveContainer" containerID="f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd" Nov 02 15:37:24.687412 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:37:24.687392 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd\": container with ID starting with f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd not found: ID does not exist" containerID="f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd" Nov 02 15:37:24.687475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.687419 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd"} err="failed to get container status \"f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd\": rpc error: code = NotFound desc = could not find container \"f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd\": container with ID starting with f1d12aa31ae0413beb2abf0122e8c5cd49242503fda2ec1eb245510a6f8bc8fd not found: ID does not exist" Nov 02 15:37:24.831957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.831880 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk"] Nov 02 15:37:24.878012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:24.877962 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa5832-6b784d46b7-cczdk"] Nov 02 15:37:26.457743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:26.457667 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b887d84-52a2-415a-99c7-4bb5de296ba3" path="/var/lib/kubelet/pods/4b887d84-52a2-415a-99c7-4bb5de296ba3/volumes" Nov 02 15:37:28.576506 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:28.576471 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:28.716566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:28.716525 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:29.450749 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:29.450708 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:33.587880 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:33.587836 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:38.158386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:38.158343 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:39.623557 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:39.623521 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:39.769475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:39.769440 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:39.914604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:39.914573 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:40.065662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:40.065623 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:40.225651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:40.225543 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:40.788414 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:40.788297 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:41.087409 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:37:41.087308 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:37:45.556257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:45.556205 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:47.044863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:47.044820 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:47.275424 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:47.275385 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:47.434496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:47.434462 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:48.276603 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:48.276572 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:49.887482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:49.887416 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:50.153227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:50.153129 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:50.785263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:50.785227 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:50.939607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:50.939567 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:51.087811 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:51.087721 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:54.320218 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:54.320182 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:54.495775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:54.495748 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:54.637928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:54.637898 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:55.675387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:55.675318 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:57.575104 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:57.575060 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:57.767997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:57.759627 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:57.903534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:57.903487 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:58.280919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:58.280745 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:58.599944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:58.599843 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:37:59.346024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:37:59.345954 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:03.364453 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:03.364380 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:05.983541 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:05.983504 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:06.122716 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:06.122678 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:06.315794 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:06.315714 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb"] Nov 02 15:38:06.315947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:06.315924 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" containerID="cri-o://19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3" gracePeriod=30 Nov 02 15:38:06.319939 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:06.319905 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:06.335828 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:06.335803 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8"] Nov 02 15:38:06.336043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:06.336024 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" containerID="cri-o://1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44" gracePeriod=30 Nov 02 15:38:06.986924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:06.986888 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:38:06.987396 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:06.986960 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:38:07.034509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:07.034477 5433 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:38:07.034690 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:07.034530 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:38:07.183324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:07.183288 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:08.118641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.118603 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj"] Nov 02 15:38:08.119473 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.119428 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" podUID="2e7ecef6-fb93-4411-b97c-3bdfe202e47f" containerName="maas-api" containerID="cri-o://d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12" gracePeriod=30 Nov 02 15:38:08.163117 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.163087 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:08.310279 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.310238 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:08.580075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.580043 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:38:08.588835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.588807 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-openshift-service-ca\") pod \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " Nov 02 15:38:08.589078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.588870 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-maas-api-tls\") pod \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " Nov 02 15:38:08.589078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.589066 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-kube-api-access-trdjh\" (UniqueName: \"kubernetes.io/projected/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-kube-api-access-trdjh\") pod \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " Nov 02 15:38:08.589226 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.589174 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-metrics-certs\") pod \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\" (UID: \"2e7ecef6-fb93-4411-b97c-3bdfe202e47f\") " Nov 02 15:38:08.589226 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.589176 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-openshift-service-ca" pod "2e7ecef6-fb93-4411-b97c-3bdfe202e47f" (UID: "2e7ecef6-fb93-4411-b97c-3bdfe202e47f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:08.589619 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.589602 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:08.591533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.591290 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-metrics-certs" pod "2e7ecef6-fb93-4411-b97c-3bdfe202e47f" (UID: "2e7ecef6-fb93-4411-b97c-3bdfe202e47f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:08.591533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.591323 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-kube-api-access-trdjh" pod "2e7ecef6-fb93-4411-b97c-3bdfe202e47f" (UID: "2e7ecef6-fb93-4411-b97c-3bdfe202e47f"). InnerVolumeSpecName "kube-api-access-trdjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:08.594383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.594356 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-maas-api-tls" pod "2e7ecef6-fb93-4411-b97c-3bdfe202e47f" (UID: "2e7ecef6-fb93-4411-b97c-3bdfe202e47f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:08.696315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.696096 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:08.696315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.696129 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:08.696315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:08.696146 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trdjh\" (UniqueName: \"kubernetes.io/projected/2e7ecef6-fb93-4411-b97c-3bdfe202e47f-kube-api-access-trdjh\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:09.045265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:09.045233 5433 generic.go:356] "Generic (PLEG): container finished" podID="2e7ecef6-fb93-4411-b97c-3bdfe202e47f" containerID="d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12" exitCode=0 Nov 02 15:38:09.045495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:09.045280 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" Nov 02 15:38:09.045495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:09.045316 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" event={"ID":"2e7ecef6-fb93-4411-b97c-3bdfe202e47f","Type":"ContainerDied","Data":"d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12"} Nov 02 15:38:09.045495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:09.045379 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj" event={"ID":"2e7ecef6-fb93-4411-b97c-3bdfe202e47f","Type":"ContainerDied","Data":"3292ec99b251eefa7aa1bd5915a59deadd8a6d448579c0fc664f0306bf5db550"} Nov 02 15:38:09.045495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:09.045409 5433 scope.go:122] "RemoveContainer" containerID="d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12" Nov 02 15:38:09.056683 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:09.056665 5433 scope.go:122] "RemoveContainer" containerID="d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12" Nov 02 15:38:09.057022 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:38:09.056956 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12\": container with ID starting with d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12 not found: ID does not exist" containerID="d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12" Nov 02 15:38:09.057022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:09.057007 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12"} err="failed to get container status \"d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12\": rpc error: code = NotFound desc = could not find container \"d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12\": container with ID starting with d5ed7025864d1cdce98b1775cfec0bb1728302f4a428aed1479dee2f4ac38a12 not found: ID does not exist" Nov 02 15:38:09.205783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:09.205751 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj"] Nov 02 15:38:09.209269 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:09.209245 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5aeac88e-77967d9754-k4snj"] Nov 02 15:38:10.055150 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:10.055118 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:10.347692 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:10.347607 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:10.504521 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:10.504478 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:10.514045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:10.514008 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7ecef6-fb93-4411-b97c-3bdfe202e47f" path="/var/lib/kubelet/pods/2e7ecef6-fb93-4411-b97c-3bdfe202e47f/volumes" Nov 02 15:38:11.015874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.015759 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:11.106725 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:38:11.106688 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:38:11.630615 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.630563 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:38:11.667941 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.667908 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-data\") pod \"8a021967-af9b-4b44-b323-04e82c9e28da\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " Nov 02 15:38:11.668125 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668069 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a021967-af9b-4b44-b323-04e82c9e28da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a021967-af9b-4b44-b323-04e82c9e28da-istio-podinfo\") pod \"8a021967-af9b-4b44-b323-04e82c9e28da\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " Nov 02 15:38:11.668192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668181 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-socket\") pod \"8a021967-af9b-4b44-b323-04e82c9e28da\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " Nov 02 15:38:11.668277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668245 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-data" pod "8a021967-af9b-4b44-b323-04e82c9e28da" (UID: "8a021967-af9b-4b44-b323-04e82c9e28da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.668418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668402 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-envoy\") pod \"8a021967-af9b-4b44-b323-04e82c9e28da\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " Nov 02 15:38:11.668491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668431 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-socket" pod "8a021967-af9b-4b44-b323-04e82c9e28da" (UID: "8a021967-af9b-4b44-b323-04e82c9e28da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.668491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668440 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-istio-token\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-istio-token\") pod \"8a021967-af9b-4b44-b323-04e82c9e28da\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " Nov 02 15:38:11.668491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668472 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-kube-api-access-k9tpq\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-kube-api-access-k9tpq\") pod \"8a021967-af9b-4b44-b323-04e82c9e28da\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " Nov 02 15:38:11.668647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668506 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a021967-af9b-4b44-b323-04e82c9e28da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a021967-af9b-4b44-b323-04e82c9e28da-istiod-ca-cert\") pod \"8a021967-af9b-4b44-b323-04e82c9e28da\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " Nov 02 15:38:11.668647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668534 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-certs\") pod \"8a021967-af9b-4b44-b323-04e82c9e28da\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " Nov 02 15:38:11.668647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668608 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-credential-socket\") pod \"8a021967-af9b-4b44-b323-04e82c9e28da\" (UID: \"8a021967-af9b-4b44-b323-04e82c9e28da\") " Nov 02 15:38:11.668896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.668870 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-certs" pod "8a021967-af9b-4b44-b323-04e82c9e28da" (UID: "8a021967-af9b-4b44-b323-04e82c9e28da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.669027 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.669001 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-credential-socket" pod "8a021967-af9b-4b44-b323-04e82c9e28da" (UID: "8a021967-af9b-4b44-b323-04e82c9e28da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.669212 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.669190 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a021967-af9b-4b44-b323-04e82c9e28da-istiod-ca-cert" pod "8a021967-af9b-4b44-b323-04e82c9e28da" (UID: "8a021967-af9b-4b44-b323-04e82c9e28da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:11.669452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.669431 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.669543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.669459 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.669543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.669475 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a021967-af9b-4b44-b323-04e82c9e28da-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.669543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.669488 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.669543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.669502 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.670622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.670593 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a021967-af9b-4b44-b323-04e82c9e28da-istio-podinfo" pod "8a021967-af9b-4b44-b323-04e82c9e28da" (UID: "8a021967-af9b-4b44-b323-04e82c9e28da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:38:11.671074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.670953 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-istio-token" pod "8a021967-af9b-4b44-b323-04e82c9e28da" (UID: "8a021967-af9b-4b44-b323-04e82c9e28da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:11.671074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.671070 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-kube-api-access-k9tpq" pod "8a021967-af9b-4b44-b323-04e82c9e28da" (UID: "8a021967-af9b-4b44-b323-04e82c9e28da"). InnerVolumeSpecName "kube-api-access-k9tpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:11.671617 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.671600 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-envoy" pod "8a021967-af9b-4b44-b323-04e82c9e28da" (UID: "8a021967-af9b-4b44-b323-04e82c9e28da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.683967 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.683946 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:38:11.770092 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770061 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-data\") pod \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " Nov 02 15:38:11.770295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770128 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-certs\") pod \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " Nov 02 15:38:11.770295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770159 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-podinfo\") pod \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " Nov 02 15:38:11.770295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770207 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-token\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-token\") pod \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " Nov 02 15:38:11.770472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770299 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-credential-socket\") pod \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " Nov 02 15:38:11.770472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770338 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-envoy\") pod \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " Nov 02 15:38:11.770472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770373 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-data" pod "b5eba7ca-7738-4cb2-a21d-dd1bee386316" (UID: "b5eba7ca-7738-4cb2-a21d-dd1bee386316"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.770472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770383 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-certs" pod "b5eba7ca-7738-4cb2-a21d-dd1bee386316" (UID: "b5eba7ca-7738-4cb2-a21d-dd1bee386316"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.770472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770429 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-socket\") pod \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " Nov 02 15:38:11.770727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770474 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istiod-ca-cert\") pod \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " Nov 02 15:38:11.770727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770538 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-kube-api-access-7kt2n\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-kube-api-access-7kt2n\") pod \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\" (UID: \"b5eba7ca-7738-4cb2-a21d-dd1bee386316\") " Nov 02 15:38:11.770946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.770922 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-socket" pod "b5eba7ca-7738-4cb2-a21d-dd1bee386316" (UID: "b5eba7ca-7738-4cb2-a21d-dd1bee386316"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.771290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.771255 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istiod-ca-cert" pod "b5eba7ca-7738-4cb2-a21d-dd1bee386316" (UID: "b5eba7ca-7738-4cb2-a21d-dd1bee386316"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:11.771528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.771506 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-credential-socket" pod "b5eba7ca-7738-4cb2-a21d-dd1bee386316" (UID: "b5eba7ca-7738-4cb2-a21d-dd1bee386316"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.772085 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.772059 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.772197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.772089 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a021967-af9b-4b44-b323-04e82c9e28da-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.772197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.772106 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.772197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.772120 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.772197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.772133 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.772197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.772147 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a021967-af9b-4b44-b323-04e82c9e28da-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.772197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.772160 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.772197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.772176 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9tpq\" (UniqueName: \"kubernetes.io/projected/8a021967-af9b-4b44-b323-04e82c9e28da-kube-api-access-k9tpq\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.772426 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.772201 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.773590 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.773168 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-envoy" pod "b5eba7ca-7738-4cb2-a21d-dd1bee386316" (UID: "b5eba7ca-7738-4cb2-a21d-dd1bee386316"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.775346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.775315 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-token" pod "b5eba7ca-7738-4cb2-a21d-dd1bee386316" (UID: "b5eba7ca-7738-4cb2-a21d-dd1bee386316"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:11.775346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.775321 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-podinfo" pod "b5eba7ca-7738-4cb2-a21d-dd1bee386316" (UID: "b5eba7ca-7738-4cb2-a21d-dd1bee386316"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:38:11.776513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.776492 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-kube-api-access-7kt2n" pod "b5eba7ca-7738-4cb2-a21d-dd1bee386316" (UID: "b5eba7ca-7738-4cb2-a21d-dd1bee386316"). InnerVolumeSpecName "kube-api-access-7kt2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:11.873605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.873529 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.873605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.873558 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.873605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.873569 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kt2n\" (UniqueName: \"kubernetes.io/projected/b5eba7ca-7738-4cb2-a21d-dd1bee386316-kube-api-access-7kt2n\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.873605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:11.873579 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5eba7ca-7738-4cb2-a21d-dd1bee386316-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:12.090574 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.090536 5433 generic.go:356] "Generic (PLEG): container finished" podID="8a021967-af9b-4b44-b323-04e82c9e28da" containerID="19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3" exitCode=0 Nov 02 15:38:12.090762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.090591 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" Nov 02 15:38:12.090762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.090621 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" event={"ID":"8a021967-af9b-4b44-b323-04e82c9e28da","Type":"ContainerDied","Data":"19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3"} Nov 02 15:38:12.090762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.090681 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb" event={"ID":"8a021967-af9b-4b44-b323-04e82c9e28da","Type":"ContainerDied","Data":"1afa6e32e74bb96d5802022b1c9f5475565a55ddef1adad0096fdf59be633556"} Nov 02 15:38:12.090762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.090713 5433 scope.go:122] "RemoveContainer" containerID="19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3" Nov 02 15:38:12.092338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.092313 5433 generic.go:356] "Generic (PLEG): container finished" podID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerID="1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44" exitCode=0 Nov 02 15:38:12.092494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.092365 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" Nov 02 15:38:12.092494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.092390 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" event={"ID":"b5eba7ca-7738-4cb2-a21d-dd1bee386316","Type":"ContainerDied","Data":"1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44"} Nov 02 15:38:12.092494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.092423 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8" event={"ID":"b5eba7ca-7738-4cb2-a21d-dd1bee386316","Type":"ContainerDied","Data":"37433187b143635b9440d765feb212dfb9b69b2d70c7dd7b7c1e35872f931f7d"} Nov 02 15:38:12.101455 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.101436 5433 scope.go:122] "RemoveContainer" containerID="19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3" Nov 02 15:38:12.101804 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:38:12.101731 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3\": container with ID starting with 19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3 not found: ID does not exist" containerID="19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3" Nov 02 15:38:12.101804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.101765 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3"} err="failed to get container status \"19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3\": rpc error: code = NotFound desc = could not find container \"19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3\": container with ID starting with 19f5636b74bd872379e788a174a40c47e64b7d61e3f2717a2f736b3c553909f3 not found: ID does not exist" Nov 02 15:38:12.101804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.101788 5433 scope.go:122] "RemoveContainer" containerID="1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44" Nov 02 15:38:12.113288 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.113269 5433 scope.go:122] "RemoveContainer" containerID="1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44" Nov 02 15:38:12.113605 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:38:12.113584 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44\": container with ID starting with 1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44 not found: ID does not exist" containerID="1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44" Nov 02 15:38:12.113690 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.113617 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44"} err="failed to get container status \"1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44\": rpc error: code = NotFound desc = could not find container \"1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44\": container with ID starting with 1f617077062fd6634ed1dd4333c328957dec7e179ab341161d5badccca90dd44 not found: ID does not exist" Nov 02 15:38:12.246291 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.246252 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8"] Nov 02 15:38:12.251392 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.251365 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-q5kb8"] Nov 02 15:38:12.352837 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.352803 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb"] Nov 02 15:38:12.357445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.357417 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-rzdgb"] Nov 02 15:38:12.457242 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.457213 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" path="/var/lib/kubelet/pods/8a021967-af9b-4b44-b323-04e82c9e28da/volumes" Nov 02 15:38:12.457688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.457673 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" path="/var/lib/kubelet/pods/b5eba7ca-7738-4cb2-a21d-dd1bee386316/volumes" Nov 02 15:38:12.705338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:12.705306 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:13.149521 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.149487 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:13.378355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.378317 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:13.554513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.554427 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:13.637255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637222 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b"] Nov 02 15:38:13.637882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637868 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" Nov 02 15:38:13.637936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637884 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" Nov 02 15:38:13.637936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637897 5433 cpu_manager.go:414] "containerMap: removing container" podUID="2e7ecef6-fb93-4411-b97c-3bdfe202e47f" containerName="maas-api" Nov 02 15:38:13.637936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637902 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7ecef6-fb93-4411-b97c-3bdfe202e47f" containerName="maas-api" Nov 02 15:38:13.637936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637924 5433 cpu_manager.go:414] "containerMap: removing container" podUID="7ba9a13a-5bd4-4d29-aa48-b10162823acf" containerName="payload-pre-processing" Nov 02 15:38:13.637936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637931 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba9a13a-5bd4-4d29-aa48-b10162823acf" containerName="payload-pre-processing" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637943 5433 cpu_manager.go:414] "containerMap: removing container" podUID="4b887d84-52a2-415a-99c7-4bb5de296ba3" containerName="payload-pre-processing" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637948 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b887d84-52a2-415a-99c7-4bb5de296ba3" containerName="payload-pre-processing" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637955 5433 cpu_manager.go:414] "containerMap: removing container" podUID="82934be9-75b5-4023-8013-2ae3c5456011" containerName="payload-processing" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637960 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82934be9-75b5-4023-8013-2ae3c5456011" containerName="payload-processing" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637990 5433 cpu_manager.go:414] "containerMap: removing container" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.637995 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.638101 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5eba7ca-7738-4cb2-a21d-dd1bee386316" containerName="istio-proxy" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.638112 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7ecef6-fb93-4411-b97c-3bdfe202e47f" containerName="maas-api" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.638119 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba9a13a-5bd4-4d29-aa48-b10162823acf" containerName="payload-pre-processing" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.638126 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b887d84-52a2-415a-99c7-4bb5de296ba3" containerName="payload-pre-processing" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.638132 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="82934be9-75b5-4023-8013-2ae3c5456011" containerName="payload-processing" Nov 02 15:38:13.638134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.638138 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a021967-af9b-4b44-b323-04e82c9e28da" containerName="istio-proxy" Nov 02 15:38:13.643246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.643224 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.697000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.695069 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f6b245-openshift-default-dockercfg-zvtn2\"" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.707896 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c978cb20-4d48-48db-a682-f8d9916a460e-istio-podinfo\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.708171 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-certs\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.708220 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-data\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.708274 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-istio-token\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.708309 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j577t\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-kube-api-access-j577t\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.708349 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-socket\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.708373 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c978cb20-4d48-48db-a682-f8d9916a460e-istiod-ca-cert\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.708431 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-credential-socket\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.708556 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-envoy\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.732202 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:13.786037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.747593 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b"] Nov 02 15:38:13.841310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.841216 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-istio-token\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.861660 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.861616 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j577t\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-kube-api-access-j577t\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.866158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.862912 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-socket\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.934293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.895602 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c978cb20-4d48-48db-a682-f8d9916a460e-istiod-ca-cert\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.934618 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.934597 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-credential-socket\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.934914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.934894 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-envoy\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.935175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.935157 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c978cb20-4d48-48db-a682-f8d9916a460e-istio-podinfo\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.935532 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.935494 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-certs\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.935708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.935690 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-data\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.936289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.895524 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-socket\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.936614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.936598 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-data\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.936763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.934243 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c978cb20-4d48-48db-a682-f8d9916a460e-istiod-ca-cert\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.937180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.907853 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:13.953042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.943433 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-envoy\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.953042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.946165 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j577t\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-kube-api-access-j577t\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.953042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.946313 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-certs\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.953042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.946318 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-credential-socket\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.953042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.950870 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-istio-token\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.953669 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.953630 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c978cb20-4d48-48db-a682-f8d9916a460e-istio-podinfo\") pod \"e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:13.956849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:13.956823 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:14.185437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:14.185395 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:14.387482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:14.387454 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b"] Nov 02 15:38:14.389172 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:38:14.389127 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc978cb20_4d48_48db_a682_f8d9916a460e.slice/crio-7a2dc96bf56d16916195b76807aff81e4726aead5661af8b24fd1b83c5f282d3 WatchSource:0}: Error finding container 7a2dc96bf56d16916195b76807aff81e4726aead5661af8b24fd1b83c5f282d3: Status 404 returned error can't find the container with id 7a2dc96bf56d16916195b76807aff81e4726aead5661af8b24fd1b83c5f282d3 Nov 02 15:38:14.392149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:14.392115 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:38:14.392247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:14.392189 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:38:14.392247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:14.392220 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:38:15.117636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:15.117602 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" event={"ID":"c978cb20-4d48-48db-a682-f8d9916a460e","Type":"ContainerStarted","Data":"b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249"} Nov 02 15:38:15.118097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:15.117655 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" event={"ID":"c978cb20-4d48-48db-a682-f8d9916a460e","Type":"ContainerStarted","Data":"7a2dc96bf56d16916195b76807aff81e4726aead5661af8b24fd1b83c5f282d3"} Nov 02 15:38:15.232758 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:15.232697 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" podStartSLOduration=2.232678584 podStartE2EDuration="2.232678584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:15.230697715 +0000 UTC m=+2398.903900910" watchObservedRunningTime="2026-11-02 15:38:15.232678584 +0000 UTC m=+2398.905881780" Nov 02 15:38:15.957493 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:15.957459 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:15.963569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:15.963544 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:15.979780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:15.979752 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j"] Nov 02 15:38:16.073395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:16.073356 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:16.150998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:16.145841 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:16.151498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:16.151325 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:38:16.222308 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:16.222211 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:16.624822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:16.624792 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:16.849210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:16.848504 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:17.042095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.042007 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4"] Nov 02 15:38:17.046624 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.046599 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:17.059621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.059590 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkr8k\" (UniqueName: \"kubernetes.io/projected/95cc3a4b-68fe-42f3-b931-70b6d1132b05-kube-api-access-bkr8k\") pod \"payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4\" (UID: \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:17.059757 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.059675 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95cc3a4b-68fe-42f3-b931-70b6d1132b05-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4\" (UID: \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:17.097560 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.097531 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-eb6bd4\"" Nov 02 15:38:17.098436 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.098201 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-eb6bd4-dockercfg-6z48l\"" Nov 02 15:38:17.101474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.101446 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4"] Nov 02 15:38:17.166000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.164824 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkr8k\" (UniqueName: \"kubernetes.io/projected/95cc3a4b-68fe-42f3-b931-70b6d1132b05-kube-api-access-bkr8k\") pod \"payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4\" (UID: \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:17.166000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.164961 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95cc3a4b-68fe-42f3-b931-70b6d1132b05-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4\" (UID: \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:17.174231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.166809 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95cc3a4b-68fe-42f3-b931-70b6d1132b05-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4\" (UID: \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:17.241998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.241783 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2"] Nov 02 15:38:17.266069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.257332 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:17.299301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.290641 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt75c\" (UniqueName: \"kubernetes.io/projected/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-kube-api-access-nt75c\") pod \"payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2\" (UID: \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:17.299301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.291644 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2\" (UID: \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:17.299301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.294197 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkr8k\" (UniqueName: \"kubernetes.io/projected/95cc3a4b-68fe-42f3-b931-70b6d1132b05-kube-api-access-bkr8k\") pod \"payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4\" (UID: \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:17.299301 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.295111 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2"] Nov 02 15:38:17.362901 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.362868 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:17.393032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.393002 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2\" (UID: \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:17.393204 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.393092 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt75c\" (UniqueName: \"kubernetes.io/projected/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-kube-api-access-nt75c\") pod \"payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2\" (UID: \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:17.393915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.393888 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2\" (UID: \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:17.452419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.452387 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt75c\" (UniqueName: \"kubernetes.io/projected/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-kube-api-access-nt75c\") pod \"payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2\" (UID: \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:17.625205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.625176 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:17.736697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.736668 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4"] Nov 02 15:38:17.740636 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:38:17.740307 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cc3a4b_68fe_42f3_b931_70b6d1132b05.slice/crio-5ac5a6f424c993c46d0a1a7bab3c2d27b2867e0ed9f47f7b2197acc49bdd0cc9 WatchSource:0}: Error finding container 5ac5a6f424c993c46d0a1a7bab3c2d27b2867e0ed9f47f7b2197acc49bdd0cc9: Status 404 returned error can't find the container with id 5ac5a6f424c993c46d0a1a7bab3c2d27b2867e0ed9f47f7b2197acc49bdd0cc9 Nov 02 15:38:17.999693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:17.999666 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2"] Nov 02 15:38:18.000888 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:38:18.000860 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1aaa7bc_3c3a_43de_8c9f_c76311fa62ca.slice/crio-a6e641de7c6212a570ba3820d02fcaef36742431267ec824be88e9fd45e85171 WatchSource:0}: Error finding container a6e641de7c6212a570ba3820d02fcaef36742431267ec824be88e9fd45e85171: Status 404 returned error can't find the container with id a6e641de7c6212a570ba3820d02fcaef36742431267ec824be88e9fd45e85171 Nov 02 15:38:18.262887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.262780 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" event={"ID":"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca","Type":"ContainerStarted","Data":"a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97"} Nov 02 15:38:18.262887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.262852 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:18.262887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.262875 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" event={"ID":"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca","Type":"ContainerStarted","Data":"a6e641de7c6212a570ba3820d02fcaef36742431267ec824be88e9fd45e85171"} Nov 02 15:38:18.264676 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.264645 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" event={"ID":"95cc3a4b-68fe-42f3-b931-70b6d1132b05","Type":"ContainerStarted","Data":"6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d"} Nov 02 15:38:18.264808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.264701 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" event={"ID":"95cc3a4b-68fe-42f3-b931-70b6d1132b05","Type":"ContainerStarted","Data":"5ac5a6f424c993c46d0a1a7bab3c2d27b2867e0ed9f47f7b2197acc49bdd0cc9"} Nov 02 15:38:18.264885 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.264865 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:18.296126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.296069 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:18.374240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.374190 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" podStartSLOduration=1.3741746369999999 podStartE2EDuration="1.374174637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:18.371757713 +0000 UTC m=+2402.044960909" watchObservedRunningTime="2026-11-02 15:38:18.374174637 +0000 UTC m=+2402.047377848" Nov 02 15:38:18.480821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.480757 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" podStartSLOduration=2.480739979 podStartE2EDuration="2.480739979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:18.479766422 +0000 UTC m=+2402.152969618" watchObservedRunningTime="2026-11-02 15:38:18.480739979 +0000 UTC m=+2402.153943170" Nov 02 15:38:18.489413 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:18.489380 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:21.555023 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:21.554955 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:21.708722 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:21.708680 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:21.857010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:21.856895 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:22.086517 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.085209 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd"] Nov 02 15:38:22.115769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.114159 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:22.187439 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.173780 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd"] Nov 02 15:38:22.203247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.202924 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f6b245\"" Nov 02 15:38:22.203247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.203134 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f6b245-dockercfg-4xmcq\"" Nov 02 15:38:22.228474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.228431 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2267b7-0f71-412b-b161-1d2425f884fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd\" (UID: \"2b2267b7-0f71-412b-b161-1d2425f884fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:22.228687 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.228664 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mh7z\" (UniqueName: \"kubernetes.io/projected/2b2267b7-0f71-412b-b161-1d2425f884fa-kube-api-access-2mh7z\") pod \"payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd\" (UID: \"2b2267b7-0f71-412b-b161-1d2425f884fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:22.330281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.330240 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2267b7-0f71-412b-b161-1d2425f884fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd\" (UID: \"2b2267b7-0f71-412b-b161-1d2425f884fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:22.330485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.330423 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mh7z\" (UniqueName: \"kubernetes.io/projected/2b2267b7-0f71-412b-b161-1d2425f884fa-kube-api-access-2mh7z\") pod \"payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd\" (UID: \"2b2267b7-0f71-412b-b161-1d2425f884fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:22.331062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.331026 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2267b7-0f71-412b-b161-1d2425f884fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd\" (UID: \"2b2267b7-0f71-412b-b161-1d2425f884fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:22.388530 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.388496 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mh7z\" (UniqueName: \"kubernetes.io/projected/2b2267b7-0f71-412b-b161-1d2425f884fa-kube-api-access-2mh7z\") pod \"payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd\" (UID: \"2b2267b7-0f71-412b-b161-1d2425f884fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:22.504263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.504227 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:22.889400 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:22.889372 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd"] Nov 02 15:38:22.891540 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:38:22.891506 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2267b7_0f71_412b_b161_1d2425f884fa.slice/crio-1e71e40ac534021be48d489043b1d5e9b8970b0146ba58a0714bc527a22fd559 WatchSource:0}: Error finding container 1e71e40ac534021be48d489043b1d5e9b8970b0146ba58a0714bc527a22fd559: Status 404 returned error can't find the container with id 1e71e40ac534021be48d489043b1d5e9b8970b0146ba58a0714bc527a22fd559 Nov 02 15:38:23.297882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:23.297782 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" event={"ID":"2b2267b7-0f71-412b-b161-1d2425f884fa","Type":"ContainerStarted","Data":"331cd79b6f42172014e192f35695112ca177a23c7d9a79bdb6ba526dd5aaf545"} Nov 02 15:38:23.298083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:23.297889 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:23.298083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:23.297912 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" event={"ID":"2b2267b7-0f71-412b-b161-1d2425f884fa","Type":"ContainerStarted","Data":"1e71e40ac534021be48d489043b1d5e9b8970b0146ba58a0714bc527a22fd559"} Nov 02 15:38:23.411034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:23.410956 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" podStartSLOduration=1.410940943 podStartE2EDuration="1.410940943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:23.409315984 +0000 UTC m=+2407.082519179" watchObservedRunningTime="2026-11-02 15:38:23.410940943 +0000 UTC m=+2407.084144138" Nov 02 15:38:27.313018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:27.312954 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:28.872163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:28.872129 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:29.028659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:29.028620 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:29.275296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:29.275267 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:38:29.275474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:29.275344 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:38:29.810748 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:29.810697 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:31.004819 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.004781 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" podUID="cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" containerName="main" containerID="cri-o://6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30" gracePeriod=45 Nov 02 15:38:31.290922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.290897 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:38:31.351032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.351000 5433 generic.go:356] "Generic (PLEG): container finished" podID="cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" containerID="6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30" exitCode=0 Nov 02 15:38:31.351165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.351034 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" event={"ID":"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88","Type":"ContainerDied","Data":"6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30"} Nov 02 15:38:31.351165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.351063 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" Nov 02 15:38:31.351165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.351085 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j" event={"ID":"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88","Type":"ContainerDied","Data":"35b747569256556eccc354313781404d25b92121bc7dd3734d87802e61653f47"} Nov 02 15:38:31.351165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.351108 5433 scope.go:122] "RemoveContainer" containerID="6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30" Nov 02 15:38:31.364805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.364788 5433 scope.go:122] "RemoveContainer" containerID="6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30" Nov 02 15:38:31.365164 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:38:31.365144 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30\": container with ID starting with 6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30 not found: ID does not exist" containerID="6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30" Nov 02 15:38:31.365258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.365176 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30"} err="failed to get container status \"6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30\": rpc error: code = NotFound desc = could not find container \"6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30\": container with ID starting with 6d8728bea893df0c77b8507eb9e38a2a3e59e28b239808675b5dd5a29136ab30 not found: ID does not exist" Nov 02 15:38:31.472488 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.472460 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tls-certs\") pod \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " Nov 02 15:38:31.472689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.472518 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-model-cache\") pod \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " Nov 02 15:38:31.472689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.472575 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tmp-dir\") pod \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " Nov 02 15:38:31.472689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.472680 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-dshm\") pod \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " Nov 02 15:38:31.472873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.472715 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-kube-api-access-dst7t\" (UniqueName: \"kubernetes.io/projected/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-kube-api-access-dst7t\") pod \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " Nov 02 15:38:31.472873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.472813 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-model-cache" pod "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" (UID: "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:31.473015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.472880 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tmp-dir" pod "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" (UID: "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:31.473015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.472916 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-home\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-home\") pod \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\" (UID: \"cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88\") " Nov 02 15:38:31.473217 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.473191 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-home" pod "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" (UID: "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:31.473575 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.473546 5433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-home\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:31.473575 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.473565 5433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-model-cache\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:31.473723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.473595 5433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tmp-dir\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:31.475022 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.474963 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-dshm" pod "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" (UID: "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:31.475184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.475165 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tls-certs" pod "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" (UID: "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:31.475226 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.475195 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-kube-api-access-dst7t" pod "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" (UID: "cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88"). InnerVolumeSpecName "kube-api-access-dst7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:31.574906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.574825 5433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-dshm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:31.574906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.574853 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dst7t\" (UniqueName: \"kubernetes.io/projected/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-kube-api-access-dst7t\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:31.574906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.574865 5433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88-tls-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:31.765816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.765783 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j"] Nov 02 15:38:31.769542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:31.769515 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5aeac88e/e2e-lifecycle-model-5aeac88e-kserve-fdc845669-2kj4j"] Nov 02 15:38:32.457810 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:32.457733 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" path="/var/lib/kubelet/pods/cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88/volumes" Nov 02 15:38:32.547673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:32.547638 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:34.200314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:34.200282 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:34.308130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:34.308103 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:38:34.470999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:34.470894 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:34.627721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:34.627616 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:35.250380 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:35.250349 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:37.533548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:37.533506 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:37.681727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:37.681688 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:39.133273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.133233 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29"] Nov 02 15:38:39.133658 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.133492 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" podUID="48a59200-a748-4b25-b837-0eefea63069a" containerName="maas-api" containerID="cri-o://4b9ec91a307f26d6daf7a0d9f9115023d5355011ef7bcb89f9af70516cf5d3a8" gracePeriod=30 Nov 02 15:38:39.233290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.233249 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:39.422324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.422065 5433 generic.go:356] "Generic (PLEG): container finished" podID="48a59200-a748-4b25-b837-0eefea63069a" containerID="4b9ec91a307f26d6daf7a0d9f9115023d5355011ef7bcb89f9af70516cf5d3a8" exitCode=0 Nov 02 15:38:39.422324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.422228 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" event={"ID":"48a59200-a748-4b25-b837-0eefea63069a","Type":"ContainerDied","Data":"4b9ec91a307f26d6daf7a0d9f9115023d5355011ef7bcb89f9af70516cf5d3a8"} Nov 02 15:38:39.472459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.472434 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:38:39.531609 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.531582 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls\") pod \"48a59200-a748-4b25-b837-0eefea63069a\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " Nov 02 15:38:39.531769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.531634 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a59200-a748-4b25-b837-0eefea63069a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a59200-a748-4b25-b837-0eefea63069a-openshift-service-ca\") pod \"48a59200-a748-4b25-b837-0eefea63069a\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " Nov 02 15:38:39.531769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.531672 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-metrics-certs\") pod \"48a59200-a748-4b25-b837-0eefea63069a\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " Nov 02 15:38:39.531843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.531775 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a59200-a748-4b25-b837-0eefea63069a-kube-api-access-md9g4\" (UniqueName: \"kubernetes.io/projected/48a59200-a748-4b25-b837-0eefea63069a-kube-api-access-md9g4\") pod \"48a59200-a748-4b25-b837-0eefea63069a\" (UID: \"48a59200-a748-4b25-b837-0eefea63069a\") " Nov 02 15:38:39.532155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.532122 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a59200-a748-4b25-b837-0eefea63069a-openshift-service-ca" pod "48a59200-a748-4b25-b837-0eefea63069a" (UID: "48a59200-a748-4b25-b837-0eefea63069a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:39.532665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.532640 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a59200-a748-4b25-b837-0eefea63069a-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:39.534298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.534259 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a59200-a748-4b25-b837-0eefea63069a-kube-api-access-md9g4" pod "48a59200-a748-4b25-b837-0eefea63069a" (UID: "48a59200-a748-4b25-b837-0eefea63069a"). InnerVolumeSpecName "kube-api-access-md9g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:39.534437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.534416 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-metrics-certs" pod "48a59200-a748-4b25-b837-0eefea63069a" (UID: "48a59200-a748-4b25-b837-0eefea63069a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:39.534527 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.534493 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls" pod "48a59200-a748-4b25-b837-0eefea63069a" (UID: "48a59200-a748-4b25-b837-0eefea63069a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:39.634465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.634433 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:39.634465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.634468 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a59200-a748-4b25-b837-0eefea63069a-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:39.634672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:39.634482 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md9g4\" (UniqueName: \"kubernetes.io/projected/48a59200-a748-4b25-b837-0eefea63069a-kube-api-access-md9g4\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:40.430585 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:40.430548 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" Nov 02 15:38:40.431073 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:40.430552 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29" event={"ID":"48a59200-a748-4b25-b837-0eefea63069a","Type":"ContainerDied","Data":"a2cd9a26ad375b9a85c16906e3e77ddd82515062a279ca1ad5f7198b54466f1a"} Nov 02 15:38:40.431073 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:40.430692 5433 scope.go:122] "RemoveContainer" containerID="4b9ec91a307f26d6daf7a0d9f9115023d5355011ef7bcb89f9af70516cf5d3a8" Nov 02 15:38:40.586068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:40.586035 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29"] Nov 02 15:38:40.632202 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:40.632170 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-822f77-657d6c79f6-jcg29"] Nov 02 15:38:40.961851 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:40.961810 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:41.012847 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:38:41.012811 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:38:41.245385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:41.245276 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:41.402393 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:41.402358 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:41.554874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:41.554782 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:41.840327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:41.840235 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2"] Nov 02 15:38:41.840481 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:41.840443 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" containerID="cri-o://14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d" gracePeriod=30 Nov 02 15:38:41.846905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:41.846875 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:42.052670 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:42.052630 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:42.457260 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:42.457229 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a59200-a748-4b25-b837-0eefea63069a" path="/var/lib/kubelet/pods/48a59200-a748-4b25-b837-0eefea63069a/volumes" Nov 02 15:38:43.701778 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:43.701742 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:45.591437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:45.591401 5433 patch_prober.go:29] interesting pod/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:38:45.591844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:45.591471 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:38:46.780058 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:46.780016 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:47.120600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.120574 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:38:47.272555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.272515 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-certs\") pod \"232e57c9-f425-4638-89df-2eadc6a7821b\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " Nov 02 15:38:47.272766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.272745 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232e57c9-f425-4638-89df-2eadc6a7821b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232e57c9-f425-4638-89df-2eadc6a7821b-istiod-ca-cert\") pod \"232e57c9-f425-4638-89df-2eadc6a7821b\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " Nov 02 15:38:47.272843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.272814 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-istio-token\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-istio-token\") pod \"232e57c9-f425-4638-89df-2eadc6a7821b\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " Nov 02 15:38:47.272843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.272830 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-certs" pod "232e57c9-f425-4638-89df-2eadc6a7821b" (UID: "232e57c9-f425-4638-89df-2eadc6a7821b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:47.272958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.272891 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-credential-socket\") pod \"232e57c9-f425-4638-89df-2eadc6a7821b\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " Nov 02 15:38:47.272958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.272921 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/232e57c9-f425-4638-89df-2eadc6a7821b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232e57c9-f425-4638-89df-2eadc6a7821b-istio-podinfo\") pod \"232e57c9-f425-4638-89df-2eadc6a7821b\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " Nov 02 15:38:47.272958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.272937 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-data\") pod \"232e57c9-f425-4638-89df-2eadc6a7821b\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " Nov 02 15:38:47.273156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.272967 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-socket\") pod \"232e57c9-f425-4638-89df-2eadc6a7821b\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " Nov 02 15:38:47.273156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273043 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-kube-api-access-8h5c5\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-kube-api-access-8h5c5\") pod \"232e57c9-f425-4638-89df-2eadc6a7821b\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " Nov 02 15:38:47.273156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273083 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-envoy\") pod \"232e57c9-f425-4638-89df-2eadc6a7821b\" (UID: \"232e57c9-f425-4638-89df-2eadc6a7821b\") " Nov 02 15:38:47.273321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273240 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-data" pod "232e57c9-f425-4638-89df-2eadc6a7821b" (UID: "232e57c9-f425-4638-89df-2eadc6a7821b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:47.273321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273254 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-credential-socket" pod "232e57c9-f425-4638-89df-2eadc6a7821b" (UID: "232e57c9-f425-4638-89df-2eadc6a7821b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:47.273321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273265 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232e57c9-f425-4638-89df-2eadc6a7821b-istiod-ca-cert" pod "232e57c9-f425-4638-89df-2eadc6a7821b" (UID: "232e57c9-f425-4638-89df-2eadc6a7821b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:47.273465 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273327 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-socket" pod "232e57c9-f425-4638-89df-2eadc6a7821b" (UID: "232e57c9-f425-4638-89df-2eadc6a7821b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:47.273777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273753 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232e57c9-f425-4638-89df-2eadc6a7821b-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.273777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273779 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.273941 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273789 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.273941 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273798 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.273941 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.273807 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.275267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.275240 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-istio-token" pod "232e57c9-f425-4638-89df-2eadc6a7821b" (UID: "232e57c9-f425-4638-89df-2eadc6a7821b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:47.275382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.275347 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-envoy" pod "232e57c9-f425-4638-89df-2eadc6a7821b" (UID: "232e57c9-f425-4638-89df-2eadc6a7821b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:47.275543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.275519 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/232e57c9-f425-4638-89df-2eadc6a7821b-istio-podinfo" pod "232e57c9-f425-4638-89df-2eadc6a7821b" (UID: "232e57c9-f425-4638-89df-2eadc6a7821b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:38:47.275632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.275595 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-kube-api-access-8h5c5" pod "232e57c9-f425-4638-89df-2eadc6a7821b" (UID: "232e57c9-f425-4638-89df-2eadc6a7821b"). InnerVolumeSpecName "kube-api-access-8h5c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:47.382142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.376503 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.382142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.376542 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232e57c9-f425-4638-89df-2eadc6a7821b-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.382142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.376568 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5c5\" (UniqueName: \"kubernetes.io/projected/232e57c9-f425-4638-89df-2eadc6a7821b-kube-api-access-8h5c5\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.382142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.376587 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232e57c9-f425-4638-89df-2eadc6a7821b-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.530651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.530617 5433 generic.go:356] "Generic (PLEG): container finished" podID="232e57c9-f425-4638-89df-2eadc6a7821b" containerID="14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d" exitCode=0 Nov 02 15:38:47.530824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.530692 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" Nov 02 15:38:47.530824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.530696 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" event={"ID":"232e57c9-f425-4638-89df-2eadc6a7821b","Type":"ContainerDied","Data":"14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d"} Nov 02 15:38:47.530824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.530762 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2" event={"ID":"232e57c9-f425-4638-89df-2eadc6a7821b","Type":"ContainerDied","Data":"82fa27539963743ba4673bc3554257dd727405977bbc37e67e71fec1e7fff5a5"} Nov 02 15:38:47.530824 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.530793 5433 scope.go:122] "RemoveContainer" containerID="14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d" Nov 02 15:38:47.541862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.541838 5433 scope.go:122] "RemoveContainer" containerID="91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb" Nov 02 15:38:47.551253 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.551222 5433 scope.go:122] "RemoveContainer" containerID="14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d" Nov 02 15:38:47.551552 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:38:47.551527 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d\": container with ID starting with 14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d not found: ID does not exist" containerID="14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d" Nov 02 15:38:47.551609 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.551563 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d"} err="failed to get container status \"14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d\": rpc error: code = NotFound desc = could not find container \"14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d\": container with ID starting with 14f63e4230f00f2b8ad0b6e986c413ceddf51459b36895f31e469f2b4b44af6d not found: ID does not exist" Nov 02 15:38:47.551609 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.551581 5433 scope.go:122] "RemoveContainer" containerID="91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb" Nov 02 15:38:47.551858 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:38:47.551837 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb\": container with ID starting with 91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb not found: ID does not exist" containerID="91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb" Nov 02 15:38:47.551926 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.551874 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb"} err="failed to get container status \"91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb\": rpc error: code = NotFound desc = could not find container \"91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb\": container with ID starting with 91c4708d78b518747a47b064d9e56bdd269e33c6420adbd609594f91bb9a6aeb not found: ID does not exist" Nov 02 15:38:47.686337 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.686251 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2"] Nov 02 15:38:47.691943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:47.691906 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-aa5832-openshift-default-6c96d678ff-6kkr2"] Nov 02 15:38:48.458721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:48.458686 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" path="/var/lib/kubelet/pods/232e57c9-f425-4638-89df-2eadc6a7821b/volumes" Nov 02 15:38:48.662349 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:48.662315 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7"] Nov 02 15:38:48.708200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:48.708165 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w"] Nov 02 15:38:48.845383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:48.845304 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:49.028522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:49.028490 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:49.232575 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:49.232524 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:51.039763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:51.039724 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:52.344475 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:52.344431 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:53.332107 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:53.326784 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:53.772159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:53.772113 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:55.961807 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:55.961765 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk"] Nov 02 15:38:56.089093 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:56.089057 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:56.241321 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:56.240896 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:38:58.175804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:38:58.175759 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:00.012933 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:00.012831 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:00.219553 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:00.219517 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:00.334015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:00.333896 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp"] Nov 02 15:39:00.334290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:00.334257 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" containerID="cri-o://4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25" gracePeriod=30 Nov 02 15:39:00.353572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:00.353538 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w"] Nov 02 15:39:00.353753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:00.353733 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" containerID="cri-o://460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2" gracePeriod=30 Nov 02 15:39:00.369209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:00.369183 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd"] Nov 02 15:39:00.369384 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:00.369365 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" containerID="cri-o://edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78" gracePeriod=30 Nov 02 15:39:00.372319 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:00.372293 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:01.876263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:01.876223 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:01.876704 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:01.876282 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:02.484777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:02.484745 5433 patch_prober.go:29] interesting pod/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 15:39:02.484962 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:02.484799 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 15:39:02.651687 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:02.651656 5433 generic.go:356] "Generic (PLEG): container finished" podID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerID="978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c" exitCode=0 Nov 02 15:39:02.651858 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:02.651728 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" event={"ID":"4c1aacac-a3f6-4b64-84ed-2c673e7edd59","Type":"ContainerDied","Data":"978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c"} Nov 02 15:39:02.652214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:02.652199 5433 scope.go:122] "RemoveContainer" containerID="978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c" Nov 02 15:39:02.653929 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:02.653897 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:02.654086 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:02.654020 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:02.654086 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:02.654073 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:03.267879 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.267842 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:03.268395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.267919 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:03.481647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.481613 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:03.664716 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.664672 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:03.698896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.698854 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" containerID="cri-o://61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed" gracePeriod=45 Nov 02 15:39:03.726452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.725578 5433 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:03.726452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.725651 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:03.730831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.730792 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" containerID="cri-o://c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7" gracePeriod=45 Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.736521 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h"] Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737517 5433 cpu_manager.go:414] "containerMap: removing container" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737538 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737557 5433 cpu_manager.go:414] "containerMap: removing container" podUID="48a59200-a748-4b25-b837-0eefea63069a" containerName="maas-api" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737566 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a59200-a748-4b25-b837-0eefea63069a" containerName="maas-api" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737588 5433 cpu_manager.go:414] "containerMap: removing container" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737597 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737648 5433 cpu_manager.go:414] "containerMap: removing container" podUID="cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" containerName="main" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737656 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" containerName="main" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737811 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737823 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a59200-a748-4b25-b837-0eefea63069a" containerName="maas-api" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737832 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="232e57c9-f425-4638-89df-2eadc6a7821b" containerName="istio-proxy" Nov 02 15:39:03.752000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.737849 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb5cee9-2cdf-4400-9b68-a20ebb9c5a88" containerName="main" Nov 02 15:39:03.766051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.765095 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" event={"ID":"4c1aacac-a3f6-4b64-84ed-2c673e7edd59","Type":"ContainerStarted","Data":"67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da"} Nov 02 15:39:03.766193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.766080 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.777297 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.777271 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h"] Nov 02 15:39:03.814684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.814657 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9ca2cf-serving-cert\"" Nov 02 15:39:03.876877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.876845 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.877711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.876937 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-metrics-certs\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.877711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.877266 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkqdw\" (UniqueName: \"kubernetes.io/projected/69375a37-c259-47f4-a4d7-f45a25a6e85b-kube-api-access-tkqdw\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.877711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.877386 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69375a37-c259-47f4-a4d7-f45a25a6e85b-openshift-service-ca\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.890660 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.890629 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:39:03.890944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.890903 5433 patch_prober.go:29] interesting pod/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 15:39:03.891092 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.890970 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 15:39:03.979087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.979011 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.979256 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.979086 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-metrics-certs\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.979256 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.979130 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkqdw\" (UniqueName: \"kubernetes.io/projected/69375a37-c259-47f4-a4d7-f45a25a6e85b-kube-api-access-tkqdw\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.979256 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.979176 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69375a37-c259-47f4-a4d7-f45a25a6e85b-openshift-service-ca\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.979256 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:03.979191 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-serving-cert: secret "maas-api-e2e-api-a-9ca2cf-serving-cert" not found Nov 02 15:39:03.979481 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:03.979277 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls podName:69375a37-c259-47f4-a4d7-f45a25a6e85b nodeName:}" failed. No retries permitted until 2026-11-02 15:39:04.479254442 +0000 UTC m=+2448.152457621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls") pod "maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" (UID: "69375a37-c259-47f4-a4d7-f45a25a6e85b") : secret "maas-api-e2e-api-a-9ca2cf-serving-cert" not found Nov 02 15:39:03.979895 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.979863 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69375a37-c259-47f4-a4d7-f45a25a6e85b-openshift-service-ca\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:03.981739 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:03.981721 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-metrics-certs\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:04.015020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.014890 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:04.174646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.174606 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:04.174646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.174632 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkqdw\" (UniqueName: \"kubernetes.io/projected/69375a37-c259-47f4-a4d7-f45a25a6e85b-kube-api-access-tkqdw\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:04.220004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.216146 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:39:04.242044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.237194 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:39:04.344006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.337330 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.391832 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tls-certs\" (UniqueName: \"kubernetes.io/secret/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tls-certs\") pod \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.391890 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-dshm\") pod \"e9044377-0838-4232-81ba-1ea22536545e\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.391925 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tmp-dir\") pod \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.392023 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-kube-api-access-mdvrx\" (UniqueName: \"kubernetes.io/projected/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-kube-api-access-mdvrx\") pod \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.392062 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-tmp-dir\") pod \"e9044377-0838-4232-81ba-1ea22536545e\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.392117 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-home\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-home\") pod \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.392162 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-model-cache\") pod \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.392228 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9044377-0838-4232-81ba-1ea22536545e-kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/e9044377-0838-4232-81ba-1ea22536545e-kube-api-access-46qm8\") pod \"e9044377-0838-4232-81ba-1ea22536545e\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.392315 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9044377-0838-4232-81ba-1ea22536545e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9044377-0838-4232-81ba-1ea22536545e-tls-certs\") pod \"e9044377-0838-4232-81ba-1ea22536545e\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.392367 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-dshm\") pod \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\" (UID: \"37cb0552-aaa7-444b-b8cf-cc1eb3b57546\") " Nov 02 15:39:04.392427 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.392413 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-model-cache\") pod \"e9044377-0838-4232-81ba-1ea22536545e\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " Nov 02 15:39:04.393219 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.392461 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-home\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-home\") pod \"e9044377-0838-4232-81ba-1ea22536545e\" (UID: \"e9044377-0838-4232-81ba-1ea22536545e\") " Nov 02 15:39:04.403999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.403162 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-home" pod "e9044377-0838-4232-81ba-1ea22536545e" (UID: "e9044377-0838-4232-81ba-1ea22536545e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.403999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.403405 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-model-cache" pod "37cb0552-aaa7-444b-b8cf-cc1eb3b57546" (UID: "37cb0552-aaa7-444b-b8cf-cc1eb3b57546"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.403999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.403680 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-tmp-dir" pod "e9044377-0838-4232-81ba-1ea22536545e" (UID: "e9044377-0838-4232-81ba-1ea22536545e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.404284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.404043 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-home" pod "37cb0552-aaa7-444b-b8cf-cc1eb3b57546" (UID: "37cb0552-aaa7-444b-b8cf-cc1eb3b57546"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.404284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.404190 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tmp-dir" pod "37cb0552-aaa7-444b-b8cf-cc1eb3b57546" (UID: "37cb0552-aaa7-444b-b8cf-cc1eb3b57546"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.409003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.406063 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-model-cache" pod "e9044377-0838-4232-81ba-1ea22536545e" (UID: "e9044377-0838-4232-81ba-1ea22536545e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.437996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.430206 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-dshm" pod "e9044377-0838-4232-81ba-1ea22536545e" (UID: "e9044377-0838-4232-81ba-1ea22536545e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.437996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.435684 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9044377-0838-4232-81ba-1ea22536545e-tls-certs" pod "e9044377-0838-4232-81ba-1ea22536545e" (UID: "e9044377-0838-4232-81ba-1ea22536545e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:04.437996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.436258 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-dshm" pod "37cb0552-aaa7-444b-b8cf-cc1eb3b57546" (UID: "37cb0552-aaa7-444b-b8cf-cc1eb3b57546"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.437996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.437085 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9044377-0838-4232-81ba-1ea22536545e-kube-api-access-46qm8" pod "e9044377-0838-4232-81ba-1ea22536545e" (UID: "e9044377-0838-4232-81ba-1ea22536545e"). InnerVolumeSpecName "kube-api-access-46qm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:04.437996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.437210 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tls-certs" pod "37cb0552-aaa7-444b-b8cf-cc1eb3b57546" (UID: "37cb0552-aaa7-444b-b8cf-cc1eb3b57546"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:04.437996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.437356 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-kube-api-access-mdvrx" pod "37cb0552-aaa7-444b-b8cf-cc1eb3b57546" (UID: "37cb0552-aaa7-444b-b8cf-cc1eb3b57546"). InnerVolumeSpecName "kube-api-access-mdvrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:04.495075 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495013 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:04.495192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495141 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdvrx\" (UniqueName: \"kubernetes.io/projected/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-kube-api-access-mdvrx\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495154 5433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-tmp-dir\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495163 5433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-home\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495171 5433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-model-cache\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495180 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/e9044377-0838-4232-81ba-1ea22536545e-kube-api-access-46qm8\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495192 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495188 5433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9044377-0838-4232-81ba-1ea22536545e-tls-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495197 5433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-dshm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495204 5433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-model-cache\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495213 5433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-home\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495222 5433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tls-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495230 5433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9044377-0838-4232-81ba-1ea22536545e-dshm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.495538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.495238 5433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37cb0552-aaa7-444b-b8cf-cc1eb3b57546-tmp-dir\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.499118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.499101 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls\") pod \"maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:04.508023 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.508000 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:04.865262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.857634 5433 generic.go:356] "Generic (PLEG): container finished" podID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerID="c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7" exitCode=0 Nov 02 15:39:04.865262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.857787 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" event={"ID":"37cb0552-aaa7-444b-b8cf-cc1eb3b57546","Type":"ContainerDied","Data":"c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7"} Nov 02 15:39:04.865262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.857824 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" event={"ID":"37cb0552-aaa7-444b-b8cf-cc1eb3b57546","Type":"ContainerDied","Data":"fbf07ba2d68414be90e2e4f471261e80c93da0209042280bacec2dc6dfb6dc56"} Nov 02 15:39:04.865262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.857846 5433 scope.go:122] "RemoveContainer" containerID="c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7" Nov 02 15:39:04.865262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.858019 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" Nov 02 15:39:04.893271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.886850 5433 generic.go:356] "Generic (PLEG): container finished" podID="e9044377-0838-4232-81ba-1ea22536545e" containerID="61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed" exitCode=0 Nov 02 15:39:04.893271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.891557 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" Nov 02 15:39:04.893271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.892185 5433 patch_prober.go:29] interesting pod/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 15:39:04.893271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.892244 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 15:39:04.893271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.892261 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" event={"ID":"e9044377-0838-4232-81ba-1ea22536545e","Type":"ContainerDied","Data":"61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed"} Nov 02 15:39:04.893271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:04.892300 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7" event={"ID":"e9044377-0838-4232-81ba-1ea22536545e","Type":"ContainerDied","Data":"2185303dd784265f1e749a09c4b1cdb33c543c91cad7499c3c218803e04a80a6"} Nov 02 15:39:05.001410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.001380 5433 scope.go:122] "RemoveContainer" containerID="c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7" Nov 02 15:39:05.002073 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:05.002045 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7\": container with ID starting with c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7 not found: ID does not exist" containerID="c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7" Nov 02 15:39:05.002264 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.002237 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7"} err="failed to get container status \"c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7\": rpc error: code = NotFound desc = could not find container \"c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7\": container with ID starting with c20f4a6ffda0348110d418f7241beec5c5434f3f04a53c42ff5a95cd07beaca7 not found: ID does not exist" Nov 02 15:39:05.002368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.002356 5433 scope.go:122] "RemoveContainer" containerID="61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed" Nov 02 15:39:05.068742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.067329 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7"] Nov 02 15:39:05.071950 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.071915 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-2-w0-61a835-kserve-65bb94dd6b-69vd7"] Nov 02 15:39:05.074879 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.074860 5433 scope.go:122] "RemoveContainer" containerID="61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed" Nov 02 15:39:05.075247 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:05.075224 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed\": container with ID starting with 61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed not found: ID does not exist" containerID="61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed" Nov 02 15:39:05.075315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.075254 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed"} err="failed to get container status \"61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed\": rpc error: code = NotFound desc = could not find container \"61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed\": container with ID starting with 61e0f3dd3d1fca2ff919fc4c5047f796544d78ecca8e0682e100b0965d2e0eed not found: ID does not exist" Nov 02 15:39:05.172150 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.172114 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w"] Nov 02 15:39:05.173987 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.173938 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:05.178619 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.178593 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/distinct-w0-61a835-kserve-66b4c796cc-cgq9w"] Nov 02 15:39:05.353073 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.353027 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h"] Nov 02 15:39:05.354862 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:39:05.354834 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69375a37_c259_47f4_a4d7_f45a25a6e85b.slice/crio-c4657c74a10938f6d9ceff56c785a3600ba6e1fb73fbe3d3e3f0a7ba8db954d9 WatchSource:0}: Error finding container c4657c74a10938f6d9ceff56c785a3600ba6e1fb73fbe3d3e3f0a7ba8db954d9: Status 404 returned error can't find the container with id c4657c74a10938f6d9ceff56c785a3600ba6e1fb73fbe3d3e3f0a7ba8db954d9 Nov 02 15:39:05.671283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.671253 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:39:05.760002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.759967 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:39:05.764691 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.764666 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:39:05.768924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.768900 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-token\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-token\") pod \"e8066dd3-914f-4f57-aa28-b5df71b7380d\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " Nov 02 15:39:05.769099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.768947 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-envoy\") pod \"e8066dd3-914f-4f57-aa28-b5df71b7380d\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " Nov 02 15:39:05.769099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.768965 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-token\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-token\") pod \"5ef077e5-f94e-4e86-8d38-69fdda900991\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " Nov 02 15:39:05.769099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769023 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-credential-socket\") pod \"5ef077e5-f94e-4e86-8d38-69fdda900991\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " Nov 02 15:39:05.769099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769051 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef077e5-f94e-4e86-8d38-69fdda900991-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef077e5-f94e-4e86-8d38-69fdda900991-istiod-ca-cert\") pod \"5ef077e5-f94e-4e86-8d38-69fdda900991\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " Nov 02 15:39:05.769099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769077 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235eff14-e669-45ea-91db-8bd3ef7298d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235eff14-e669-45ea-91db-8bd3ef7298d2-istiod-ca-cert\") pod \"235eff14-e669-45ea-91db-8bd3ef7298d2\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " Nov 02 15:39:05.769099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769099 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-socket\") pod \"235eff14-e669-45ea-91db-8bd3ef7298d2\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " Nov 02 15:39:05.769431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769118 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-data\") pod \"5ef077e5-f94e-4e86-8d38-69fdda900991\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " Nov 02 15:39:05.769431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769168 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-credential-socket\") pod \"235eff14-e669-45ea-91db-8bd3ef7298d2\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " Nov 02 15:39:05.769431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769189 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-certs\") pod \"5ef077e5-f94e-4e86-8d38-69fdda900991\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " Nov 02 15:39:05.769431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769221 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-podinfo\") pod \"235eff14-e669-45ea-91db-8bd3ef7298d2\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " Nov 02 15:39:05.769431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769254 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-token\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-token\") pod \"235eff14-e669-45ea-91db-8bd3ef7298d2\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " Nov 02 15:39:05.769431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769296 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-podinfo\") pod \"5ef077e5-f94e-4e86-8d38-69fdda900991\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " Nov 02 15:39:05.769431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769326 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-envoy\") pod \"5ef077e5-f94e-4e86-8d38-69fdda900991\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " Nov 02 15:39:05.769431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769403 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-kube-api-access-77647\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-kube-api-access-77647\") pod \"e8066dd3-914f-4f57-aa28-b5df71b7380d\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769459 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-socket\") pod \"e8066dd3-914f-4f57-aa28-b5df71b7380d\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769515 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-podinfo\") pod \"e8066dd3-914f-4f57-aa28-b5df71b7380d\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769542 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-socket" pod "235eff14-e669-45ea-91db-8bd3ef7298d2" (UID: "235eff14-e669-45ea-91db-8bd3ef7298d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769569 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-kube-api-access-v59xj\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-kube-api-access-v59xj\") pod \"5ef077e5-f94e-4e86-8d38-69fdda900991\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769599 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-certs\") pod \"235eff14-e669-45ea-91db-8bd3ef7298d2\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769634 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8066dd3-914f-4f57-aa28-b5df71b7380d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8066dd3-914f-4f57-aa28-b5df71b7380d-istiod-ca-cert\") pod \"e8066dd3-914f-4f57-aa28-b5df71b7380d\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769652 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-credential-socket" pod "5ef077e5-f94e-4e86-8d38-69fdda900991" (UID: "5ef077e5-f94e-4e86-8d38-69fdda900991"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769677 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-envoy\") pod \"235eff14-e669-45ea-91db-8bd3ef7298d2\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769779 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef077e5-f94e-4e86-8d38-69fdda900991-istiod-ca-cert" pod "5ef077e5-f94e-4e86-8d38-69fdda900991" (UID: "5ef077e5-f94e-4e86-8d38-69fdda900991"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:05.769839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769771 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-credential-socket\") pod \"e8066dd3-914f-4f57-aa28-b5df71b7380d\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " Nov 02 15:39:05.770348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769910 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-socket" pod "e8066dd3-914f-4f57-aa28-b5df71b7380d" (UID: "e8066dd3-914f-4f57-aa28-b5df71b7380d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.770348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769969 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-credential-socket" pod "e8066dd3-914f-4f57-aa28-b5df71b7380d" (UID: "e8066dd3-914f-4f57-aa28-b5df71b7380d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.770348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.769992 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-data\") pod \"235eff14-e669-45ea-91db-8bd3ef7298d2\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " Nov 02 15:39:05.770348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.770050 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-kube-api-access-bkbhr\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-kube-api-access-bkbhr\") pod \"235eff14-e669-45ea-91db-8bd3ef7298d2\" (UID: \"235eff14-e669-45ea-91db-8bd3ef7298d2\") " Nov 02 15:39:05.770348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.770103 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-certs\") pod \"e8066dd3-914f-4f57-aa28-b5df71b7380d\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " Nov 02 15:39:05.770348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.770133 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-socket\") pod \"5ef077e5-f94e-4e86-8d38-69fdda900991\" (UID: \"5ef077e5-f94e-4e86-8d38-69fdda900991\") " Nov 02 15:39:05.770348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.770165 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-data\") pod \"e8066dd3-914f-4f57-aa28-b5df71b7380d\" (UID: \"e8066dd3-914f-4f57-aa28-b5df71b7380d\") " Nov 02 15:39:05.770694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.770512 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235eff14-e669-45ea-91db-8bd3ef7298d2-istiod-ca-cert" pod "235eff14-e669-45ea-91db-8bd3ef7298d2" (UID: "235eff14-e669-45ea-91db-8bd3ef7298d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:05.770755 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.770714 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-data" pod "e8066dd3-914f-4f57-aa28-b5df71b7380d" (UID: "e8066dd3-914f-4f57-aa28-b5df71b7380d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.771096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.770901 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-data" pod "235eff14-e669-45ea-91db-8bd3ef7298d2" (UID: "235eff14-e669-45ea-91db-8bd3ef7298d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.771096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771051 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.771096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771076 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef077e5-f94e-4e86-8d38-69fdda900991-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.771096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771091 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/235eff14-e669-45ea-91db-8bd3ef7298d2-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.771373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771098 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-credential-socket" pod "235eff14-e669-45ea-91db-8bd3ef7298d2" (UID: "235eff14-e669-45ea-91db-8bd3ef7298d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.771373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771106 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.771373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771147 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.771373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771164 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.771373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771179 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.771373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771195 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.771373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771306 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-data" pod "5ef077e5-f94e-4e86-8d38-69fdda900991" (UID: "5ef077e5-f94e-4e86-8d38-69fdda900991"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.771790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.771411 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-certs" pod "235eff14-e669-45ea-91db-8bd3ef7298d2" (UID: "235eff14-e669-45ea-91db-8bd3ef7298d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.772476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.772447 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8066dd3-914f-4f57-aa28-b5df71b7380d-istiod-ca-cert" pod "e8066dd3-914f-4f57-aa28-b5df71b7380d" (UID: "e8066dd3-914f-4f57-aa28-b5df71b7380d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:05.772722 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.772696 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-certs" pod "5ef077e5-f94e-4e86-8d38-69fdda900991" (UID: "5ef077e5-f94e-4e86-8d38-69fdda900991"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.772841 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.772817 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-certs" pod "e8066dd3-914f-4f57-aa28-b5df71b7380d" (UID: "e8066dd3-914f-4f57-aa28-b5df71b7380d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.773718 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.773693 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-socket" pod "5ef077e5-f94e-4e86-8d38-69fdda900991" (UID: "5ef077e5-f94e-4e86-8d38-69fdda900991"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.774724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.774692 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-envoy" pod "5ef077e5-f94e-4e86-8d38-69fdda900991" (UID: "5ef077e5-f94e-4e86-8d38-69fdda900991"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.774846 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.774775 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-kube-api-access-v59xj" pod "5ef077e5-f94e-4e86-8d38-69fdda900991" (UID: "5ef077e5-f94e-4e86-8d38-69fdda900991"). InnerVolumeSpecName "kube-api-access-v59xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.775283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.775252 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-kube-api-access-77647" pod "e8066dd3-914f-4f57-aa28-b5df71b7380d" (UID: "e8066dd3-914f-4f57-aa28-b5df71b7380d"). InnerVolumeSpecName "kube-api-access-77647". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.775814 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.775788 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-podinfo" pod "235eff14-e669-45ea-91db-8bd3ef7298d2" (UID: "235eff14-e669-45ea-91db-8bd3ef7298d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:05.776868 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.776841 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-envoy" pod "235eff14-e669-45ea-91db-8bd3ef7298d2" (UID: "235eff14-e669-45ea-91db-8bd3ef7298d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.776960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.776938 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-kube-api-access-bkbhr" pod "235eff14-e669-45ea-91db-8bd3ef7298d2" (UID: "235eff14-e669-45ea-91db-8bd3ef7298d2"). InnerVolumeSpecName "kube-api-access-bkbhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.777469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.777331 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-token" pod "235eff14-e669-45ea-91db-8bd3ef7298d2" (UID: "235eff14-e669-45ea-91db-8bd3ef7298d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.777469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.777368 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-token" pod "5ef077e5-f94e-4e86-8d38-69fdda900991" (UID: "5ef077e5-f94e-4e86-8d38-69fdda900991"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.777469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.777439 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-token" pod "e8066dd3-914f-4f57-aa28-b5df71b7380d" (UID: "e8066dd3-914f-4f57-aa28-b5df71b7380d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.778197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.778069 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-podinfo" pod "5ef077e5-f94e-4e86-8d38-69fdda900991" (UID: "5ef077e5-f94e-4e86-8d38-69fdda900991"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:05.778353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.778267 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-podinfo" pod "e8066dd3-914f-4f57-aa28-b5df71b7380d" (UID: "e8066dd3-914f-4f57-aa28-b5df71b7380d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:05.779401 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.779234 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-envoy" pod "e8066dd3-914f-4f57-aa28-b5df71b7380d" (UID: "e8066dd3-914f-4f57-aa28-b5df71b7380d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.872516 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872483 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872516 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872511 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77647\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-kube-api-access-77647\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872516 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872521 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872516 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872530 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v59xj\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-kube-api-access-v59xj\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872539 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872548 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8066dd3-914f-4f57-aa28-b5df71b7380d-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872556 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872565 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkbhr\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-kube-api-access-bkbhr\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872575 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872584 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872592 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872602 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8066dd3-914f-4f57-aa28-b5df71b7380d-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872610 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872618 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872626 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/235eff14-e669-45ea-91db-8bd3ef7298d2-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872634 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef077e5-f94e-4e86-8d38-69fdda900991-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872642 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872649 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235eff14-e669-45ea-91db-8bd3ef7298d2-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.872784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.872656 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef077e5-f94e-4e86-8d38-69fdda900991-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.892215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.892135 5433 patch_prober.go:29] interesting pod/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 15:39:05.892215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.892181 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 15:39:05.901341 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.901310 5433 generic.go:356] "Generic (PLEG): container finished" podID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerID="edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78" exitCode=0 Nov 02 15:39:05.901488 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.901360 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" Nov 02 15:39:05.901488 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.901390 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" event={"ID":"235eff14-e669-45ea-91db-8bd3ef7298d2","Type":"ContainerDied","Data":"edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78"} Nov 02 15:39:05.901488 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.901451 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd" event={"ID":"235eff14-e669-45ea-91db-8bd3ef7298d2","Type":"ContainerDied","Data":"795854b853c561f3fa06a966ace8519567c361ed80e8634a122f91c6a673dcc4"} Nov 02 15:39:05.901488 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.901484 5433 scope.go:122] "RemoveContainer" containerID="edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78" Nov 02 15:39:05.902866 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.902817 5433 generic.go:356] "Generic (PLEG): container finished" podID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerID="460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2" exitCode=0 Nov 02 15:39:05.902955 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.902881 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" Nov 02 15:39:05.902955 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.902912 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" event={"ID":"e8066dd3-914f-4f57-aa28-b5df71b7380d","Type":"ContainerDied","Data":"460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2"} Nov 02 15:39:05.903106 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.902969 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w" event={"ID":"e8066dd3-914f-4f57-aa28-b5df71b7380d","Type":"ContainerDied","Data":"e3e9f957c6b56feecd8ec376b407e39285013a8eef01fd7e8abb70110d97d4d2"} Nov 02 15:39:05.904370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.904344 5433 generic.go:356] "Generic (PLEG): container finished" podID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerID="4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25" exitCode=0 Nov 02 15:39:05.904511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.904387 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" Nov 02 15:39:05.904511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.904435 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" event={"ID":"5ef077e5-f94e-4e86-8d38-69fdda900991","Type":"ContainerDied","Data":"4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25"} Nov 02 15:39:05.904511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.904489 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp" event={"ID":"5ef077e5-f94e-4e86-8d38-69fdda900991","Type":"ContainerDied","Data":"58730d99fc64ca7101fa5d00837389eb0872db2e3f643e33b621f8b98397c82c"} Nov 02 15:39:05.906277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.906194 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" event={"ID":"69375a37-c259-47f4-a4d7-f45a25a6e85b","Type":"ContainerStarted","Data":"c4657c74a10938f6d9ceff56c785a3600ba6e1fb73fbe3d3e3f0a7ba8db954d9"} Nov 02 15:39:05.949761 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.949735 5433 scope.go:122] "RemoveContainer" containerID="edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78" Nov 02 15:39:05.950180 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:05.950153 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78\": container with ID starting with edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78 not found: ID does not exist" containerID="edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78" Nov 02 15:39:05.950289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.950190 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78"} err="failed to get container status \"edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78\": rpc error: code = NotFound desc = could not find container \"edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78\": container with ID starting with edd2090a5f858d617bce63a43381cae5c8a3aca1dd2088d9fc0f4e9032413d78 not found: ID does not exist" Nov 02 15:39:05.950289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.950240 5433 scope.go:122] "RemoveContainer" containerID="460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2" Nov 02 15:39:05.960634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.960608 5433 scope.go:122] "RemoveContainer" containerID="460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2" Nov 02 15:39:05.960942 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:05.960918 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2\": container with ID starting with 460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2 not found: ID does not exist" containerID="460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2" Nov 02 15:39:05.961062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.960952 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2"} err="failed to get container status \"460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2\": rpc error: code = NotFound desc = could not find container \"460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2\": container with ID starting with 460ef3ec71eb03fe1da0556399b744f386c1679c2ec401760a3271b8d0c8d1d2 not found: ID does not exist" Nov 02 15:39:05.961062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.960990 5433 scope.go:122] "RemoveContainer" containerID="4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25" Nov 02 15:39:05.977915 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.977890 5433 scope.go:122] "RemoveContainer" containerID="4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25" Nov 02 15:39:05.978225 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:05.978202 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25\": container with ID starting with 4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25 not found: ID does not exist" containerID="4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25" Nov 02 15:39:05.978303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:05.978235 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25"} err="failed to get container status \"4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25\": rpc error: code = NotFound desc = could not find container \"4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25\": container with ID starting with 4089d4e50d8c1ab59079248b0de13e543093ebcd8a46dbb6678b115f33f3db25 not found: ID does not exist" Nov 02 15:39:06.150772 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.150681 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w"] Nov 02 15:39:06.154960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.154931 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-tbz5w"] Nov 02 15:39:06.257246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.257210 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp"] Nov 02 15:39:06.261027 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.260997 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-ww6jp"] Nov 02 15:39:06.372296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.372262 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd"] Nov 02 15:39:06.377296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.377270 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-m8gbd"] Nov 02 15:39:06.458898 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.458824 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" path="/var/lib/kubelet/pods/235eff14-e669-45ea-91db-8bd3ef7298d2/volumes" Nov 02 15:39:06.459424 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.459406 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" path="/var/lib/kubelet/pods/37cb0552-aaa7-444b-b8cf-cc1eb3b57546/volumes" Nov 02 15:39:06.459873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.459860 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" path="/var/lib/kubelet/pods/5ef077e5-f94e-4e86-8d38-69fdda900991/volumes" Nov 02 15:39:06.460691 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.460674 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" path="/var/lib/kubelet/pods/e8066dd3-914f-4f57-aa28-b5df71b7380d/volumes" Nov 02 15:39:06.461100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.461087 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9044377-0838-4232-81ba-1ea22536545e" path="/var/lib/kubelet/pods/e9044377-0838-4232-81ba-1ea22536545e/volumes" Nov 02 15:39:06.898335 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.898292 5433 patch_prober.go:29] interesting pod/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 15:39:06.898515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.898342 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 15:39:06.944940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.944886 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" event={"ID":"69375a37-c259-47f4-a4d7-f45a25a6e85b","Type":"ContainerStarted","Data":"092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab"} Nov 02 15:39:06.945160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:06.945019 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:07.072913 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:07.072853 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" podStartSLOduration=3.157629682 podStartE2EDuration="4.072840767s" totalImagesPullingTime="915.211085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:07.069782869 +0000 UTC m=+2450.742986075" watchObservedRunningTime="2026-11-02 15:39:07.072840767 +0000 UTC m=+2450.746043963" Nov 02 15:39:07.891160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:07.891126 5433 patch_prober.go:29] interesting pod/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 15:39:07.891594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:07.891184 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 15:39:08.952112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:08.952082 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:39:08.952784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:08.952763 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:39:08.954176 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:08.954150 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:39:09.964270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:09.964232 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:10.983564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:10.983507 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" containerID="cri-o://9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629" gracePeriod=45 Nov 02 15:39:11.013063 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:11.013019 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:39:11.260281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.260256 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:39:11.340005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.339939 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-dshm\") pod \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " Nov 02 15:39:11.340191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.340036 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-model-cache\") pod \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " Nov 02 15:39:11.340191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.340094 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tls-certs\") pod \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " Nov 02 15:39:11.340191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.340186 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tmp-dir\") pod \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " Nov 02 15:39:11.340351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.340227 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1972a793-bbcf-407c-86d1-bfaa67eddcb8-kube-api-access-mdg9d\" (UniqueName: \"kubernetes.io/projected/1972a793-bbcf-407c-86d1-bfaa67eddcb8-kube-api-access-mdg9d\") pod \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " Nov 02 15:39:11.340351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.340292 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-model-cache" pod "1972a793-bbcf-407c-86d1-bfaa67eddcb8" (UID: "1972a793-bbcf-407c-86d1-bfaa67eddcb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.340452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.340401 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-home\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-home\") pod \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\" (UID: \"1972a793-bbcf-407c-86d1-bfaa67eddcb8\") " Nov 02 15:39:11.340505 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.340476 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tmp-dir" pod "1972a793-bbcf-407c-86d1-bfaa67eddcb8" (UID: "1972a793-bbcf-407c-86d1-bfaa67eddcb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.340662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.340637 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-home" pod "1972a793-bbcf-407c-86d1-bfaa67eddcb8" (UID: "1972a793-bbcf-407c-86d1-bfaa67eddcb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.341051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.341026 5433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tmp-dir\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.341118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.341055 5433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-home\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.341118 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.341070 5433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-model-cache\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.342303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.342276 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-dshm" pod "1972a793-bbcf-407c-86d1-bfaa67eddcb8" (UID: "1972a793-bbcf-407c-86d1-bfaa67eddcb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.342303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.342286 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tls-certs" pod "1972a793-bbcf-407c-86d1-bfaa67eddcb8" (UID: "1972a793-bbcf-407c-86d1-bfaa67eddcb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:11.342440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.342380 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1972a793-bbcf-407c-86d1-bfaa67eddcb8-kube-api-access-mdg9d" pod "1972a793-bbcf-407c-86d1-bfaa67eddcb8" (UID: "1972a793-bbcf-407c-86d1-bfaa67eddcb8"). InnerVolumeSpecName "kube-api-access-mdg9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:11.443182 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.443108 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdg9d\" (UniqueName: \"kubernetes.io/projected/1972a793-bbcf-407c-86d1-bfaa67eddcb8-kube-api-access-mdg9d\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.443182 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.443147 5433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1972a793-bbcf-407c-86d1-bfaa67eddcb8-dshm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.443182 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.443162 5433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1972a793-bbcf-407c-86d1-bfaa67eddcb8-tls-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.982755 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.982721 5433 generic.go:356] "Generic (PLEG): container finished" podID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerID="9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629" exitCode=0 Nov 02 15:39:11.982948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.982767 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" Nov 02 15:39:11.982948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.982805 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" event={"ID":"1972a793-bbcf-407c-86d1-bfaa67eddcb8","Type":"ContainerDied","Data":"9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629"} Nov 02 15:39:11.982948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.982850 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk" event={"ID":"1972a793-bbcf-407c-86d1-bfaa67eddcb8","Type":"ContainerDied","Data":"b83d5053d69513b220d9801afbe521a47146175f70e9e227ca82ef8fdde6a9cb"} Nov 02 15:39:11.982948 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.982867 5433 scope.go:122] "RemoveContainer" containerID="9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629" Nov 02 15:39:11.993799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.993629 5433 scope.go:122] "RemoveContainer" containerID="9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629" Nov 02 15:39:11.994052 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:11.993869 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629\": container with ID starting with 9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629 not found: ID does not exist" containerID="9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629" Nov 02 15:39:11.994052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:11.993892 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629"} err="failed to get container status \"9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629\": rpc error: code = NotFound desc = could not find container \"9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629\": container with ID starting with 9548856dbd13b58a67d12c887d84ff2f2267cb2bccf8aada92ab9194aa3e4629 not found: ID does not exist" Nov 02 15:39:12.137573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:12.137538 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk"] Nov 02 15:39:12.183547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:12.183509 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-2-w3-71d1eb-kserve-54b48ddc45-s6wtk"] Nov 02 15:39:12.457570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:12.457536 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" path="/var/lib/kubelet/pods/1972a793-bbcf-407c-86d1-bfaa67eddcb8/volumes" Nov 02 15:39:12.959183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:12.959151 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:39:15.362769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:15.362731 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:20.190175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:20.190134 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:24.936708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.936669 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86"] Nov 02 15:39:24.937303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937289 5433 cpu_manager.go:414] "containerMap: removing container" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" Nov 02 15:39:24.937348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937304 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" Nov 02 15:39:24.937348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937316 5433 cpu_manager.go:414] "containerMap: removing container" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" Nov 02 15:39:24.937348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937321 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" Nov 02 15:39:24.937348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937331 5433 cpu_manager.go:414] "containerMap: removing container" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" Nov 02 15:39:24.937348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937337 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" Nov 02 15:39:24.937348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937343 5433 cpu_manager.go:414] "containerMap: removing container" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" Nov 02 15:39:24.937348 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937348 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937361 5433 cpu_manager.go:414] "containerMap: removing container" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937366 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937386 5433 cpu_manager.go:414] "containerMap: removing container" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937391 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937488 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9044377-0838-4232-81ba-1ea22536545e" containerName="main" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937497 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="235eff14-e669-45ea-91db-8bd3ef7298d2" containerName="istio-proxy" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937504 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef077e5-f94e-4e86-8d38-69fdda900991" containerName="istio-proxy" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937511 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cb0552-aaa7-444b-b8cf-cc1eb3b57546" containerName="main" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937517 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1972a793-bbcf-407c-86d1-bfaa67eddcb8" containerName="main" Nov 02 15:39:24.937573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.937526 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8066dd3-914f-4f57-aa28-b5df71b7380d" containerName="istio-proxy" Nov 02 15:39:24.950490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.950464 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:24.997918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.997877 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86"] Nov 02 15:39:24.999747 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:24.999715 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-eb6bd4-serving-cert\"" Nov 02 15:39:25.117692 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.117660 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-metrics-certs\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.117872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.117712 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f969b30-d040-4ae8-afcb-3b380e4b14fd-openshift-service-ca\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.117872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.117788 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-maas-api-tls\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.117872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.117847 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtz56\" (UniqueName: \"kubernetes.io/projected/9f969b30-d040-4ae8-afcb-3b380e4b14fd-kube-api-access-mtz56\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.223044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.220393 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-metrics-certs\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.223044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.220481 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f969b30-d040-4ae8-afcb-3b380e4b14fd-openshift-service-ca\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.223044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.220519 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-maas-api-tls\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.223044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.220581 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtz56\" (UniqueName: \"kubernetes.io/projected/9f969b30-d040-4ae8-afcb-3b380e4b14fd-kube-api-access-mtz56\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.223044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.222092 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:25.223044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.222879 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f969b30-d040-4ae8-afcb-3b380e4b14fd-openshift-service-ca\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.242694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.242644 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-metrics-certs\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.243271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.243241 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-maas-api-tls\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.279053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.279019 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtz56\" (UniqueName: \"kubernetes.io/projected/9f969b30-d040-4ae8-afcb-3b380e4b14fd-kube-api-access-mtz56\") pod \"maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.383363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.382700 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:25.564920 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.564835 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:25.964124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:25.964099 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86"] Nov 02 15:39:25.966244 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:39:25.966220 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f969b30_d040_4ae8_afcb_3b380e4b14fd.slice/crio-c1f08a4bb1bf8b75ce74819dda60d3754be58a0972b52345cdcdfceeaa8bd23d WatchSource:0}: Error finding container c1f08a4bb1bf8b75ce74819dda60d3754be58a0972b52345cdcdfceeaa8bd23d: Status 404 returned error can't find the container with id c1f08a4bb1bf8b75ce74819dda60d3754be58a0972b52345cdcdfceeaa8bd23d Nov 02 15:39:26.074112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:26.074076 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" event={"ID":"9f969b30-d040-4ae8-afcb-3b380e4b14fd","Type":"ContainerStarted","Data":"c1f08a4bb1bf8b75ce74819dda60d3754be58a0972b52345cdcdfceeaa8bd23d"} Nov 02 15:39:27.078544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:27.078500 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:27.083901 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:27.083873 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" event={"ID":"9f969b30-d040-4ae8-afcb-3b380e4b14fd","Type":"ContainerStarted","Data":"405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969"} Nov 02 15:39:27.084098 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:27.084011 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:27.248601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:27.248547 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:27.591150 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:27.591068 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:30.199018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:30.198926 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:30.820576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:30.820485 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:31.418115 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:31.418064 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:39:31.558095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:31.558048 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:39:31.708191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:31.708080 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:32.095052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:32.090835 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" podStartSLOduration=7.206330261 podStartE2EDuration="8.09079695s" totalImagesPullingTime="884.466689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:27.218443677 +0000 UTC m=+2470.891646873" watchObservedRunningTime="2026-11-02 15:39:32.09079695 +0000 UTC m=+2475.764000146" Nov 02 15:39:32.095052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:32.091270 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv"] Nov 02 15:39:32.095052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:32.091592 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" containerID="cri-o://d3f205f7d8d9b5ff08a512640270d7ab76d7f4a4755fb8aab492065c1eab4478" gracePeriod=30 Nov 02 15:39:32.142340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:32.142279 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb"] Nov 02 15:39:32.142989 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:32.142889 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" containerID="cri-o://a49d3d2105fb431182dcb8c735e13c8f2c62070910b23a20e20511834a85a622" gracePeriod=30 Nov 02 15:39:32.150206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:32.150170 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5"] Nov 02 15:39:32.150680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:32.150606 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" containerID="cri-o://f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c" gracePeriod=30 Nov 02 15:39:32.296270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:32.295155 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb"] Nov 02 15:39:32.296270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:32.295408 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" containerID="cri-o://9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9" gracePeriod=30 Nov 02 15:39:33.096794 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:33.096761 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:39:34.276922 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:34.276884 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8"] Nov 02 15:39:34.277405 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:34.277158 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" containerID="cri-o://dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f" gracePeriod=30 Nov 02 15:39:35.437175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:35.437136 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:36.127175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:36.127136 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:36.355715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:36.355683 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:36.453144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:36.452490 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jhvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:36.453144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:36.452621 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:36.757359 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:36.757285 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:37.336732 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.336702 5433 generic.go:356] "Generic (PLEG): container finished" podID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerID="a49d3d2105fb431182dcb8c735e13c8f2c62070910b23a20e20511834a85a622" exitCode=0 Nov 02 15:39:37.336912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.336853 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" event={"ID":"3fad2e2e-4765-463c-8062-fa51ddde66db","Type":"ContainerDied","Data":"a49d3d2105fb431182dcb8c735e13c8f2c62070910b23a20e20511834a85a622"} Nov 02 15:39:37.338600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.338574 5433 generic.go:356] "Generic (PLEG): container finished" podID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerID="d3f205f7d8d9b5ff08a512640270d7ab76d7f4a4755fb8aab492065c1eab4478" exitCode=0 Nov 02 15:39:37.338767 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.338629 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" event={"ID":"ce9d0b06-cc52-424e-85b4-9d53582d127c","Type":"ContainerDied","Data":"d3f205f7d8d9b5ff08a512640270d7ab76d7f4a4755fb8aab492065c1eab4478"} Nov 02 15:39:37.499940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.492231 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:39:37.504494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.504470 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:39:37.573160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.573126 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:39:37.587259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587233 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-podinfo\") pod \"ce9d0b06-cc52-424e-85b4-9d53582d127c\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " Nov 02 15:39:37.587410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587296 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-certs\") pod \"ce9d0b06-cc52-424e-85b4-9d53582d127c\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " Nov 02 15:39:37.587410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587359 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fad2e2e-4765-463c-8062-fa51ddde66db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fad2e2e-4765-463c-8062-fa51ddde66db-istiod-ca-cert\") pod \"3fad2e2e-4765-463c-8062-fa51ddde66db\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " Nov 02 15:39:37.587410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587404 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-envoy\") pod \"ce9d0b06-cc52-424e-85b4-9d53582d127c\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " Nov 02 15:39:37.587589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587485 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-data\") pod \"ce9d0b06-cc52-424e-85b4-9d53582d127c\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " Nov 02 15:39:37.587589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587516 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-envoy\") pod \"3fad2e2e-4765-463c-8062-fa51ddde66db\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " Nov 02 15:39:37.587693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587585 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-socket\") pod \"3fad2e2e-4765-463c-8062-fa51ddde66db\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " Nov 02 15:39:37.587693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587638 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9d0b06-cc52-424e-85b4-9d53582d127c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9d0b06-cc52-424e-85b4-9d53582d127c-istiod-ca-cert\") pod \"ce9d0b06-cc52-424e-85b4-9d53582d127c\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " Nov 02 15:39:37.587693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587563 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-certs" pod "ce9d0b06-cc52-424e-85b4-9d53582d127c" (UID: "ce9d0b06-cc52-424e-85b4-9d53582d127c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.587846 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587716 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-token\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-token\") pod \"ce9d0b06-cc52-424e-85b4-9d53582d127c\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " Nov 02 15:39:37.587846 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587760 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-socket\") pod \"ce9d0b06-cc52-424e-85b4-9d53582d127c\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " Nov 02 15:39:37.587846 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587811 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-socket" pod "3fad2e2e-4765-463c-8062-fa51ddde66db" (UID: "3fad2e2e-4765-463c-8062-fa51ddde66db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.588050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587841 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fad2e2e-4765-463c-8062-fa51ddde66db-istiod-ca-cert" pod "3fad2e2e-4765-463c-8062-fa51ddde66db" (UID: "3fad2e2e-4765-463c-8062-fa51ddde66db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:37.588050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587848 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-credential-socket\") pod \"3fad2e2e-4765-463c-8062-fa51ddde66db\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " Nov 02 15:39:37.588050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587905 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-podinfo\") pod \"3fad2e2e-4765-463c-8062-fa51ddde66db\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " Nov 02 15:39:37.588211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588101 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-credential-socket" pod "3fad2e2e-4765-463c-8062-fa51ddde66db" (UID: "3fad2e2e-4765-463c-8062-fa51ddde66db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.588211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588129 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-data" pod "ce9d0b06-cc52-424e-85b4-9d53582d127c" (UID: "ce9d0b06-cc52-424e-85b4-9d53582d127c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.587962 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-kube-api-access-pzfxz\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-kube-api-access-pzfxz\") pod \"ce9d0b06-cc52-424e-85b4-9d53582d127c\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588370 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9d0b06-cc52-424e-85b4-9d53582d127c-istiod-ca-cert" pod "ce9d0b06-cc52-424e-85b4-9d53582d127c" (UID: "ce9d0b06-cc52-424e-85b4-9d53582d127c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588441 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-credential-socket\") pod \"ce9d0b06-cc52-424e-85b4-9d53582d127c\" (UID: \"ce9d0b06-cc52-424e-85b4-9d53582d127c\") " Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588469 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-data\") pod \"3fad2e2e-4765-463c-8062-fa51ddde66db\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588502 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-kube-api-access-9bgbt\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-kube-api-access-9bgbt\") pod \"3fad2e2e-4765-463c-8062-fa51ddde66db\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588527 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-socket" pod "ce9d0b06-cc52-424e-85b4-9d53582d127c" (UID: "ce9d0b06-cc52-424e-85b4-9d53582d127c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588544 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-token\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-token\") pod \"3fad2e2e-4765-463c-8062-fa51ddde66db\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588585 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-certs\") pod \"3fad2e2e-4765-463c-8062-fa51ddde66db\" (UID: \"3fad2e2e-4765-463c-8062-fa51ddde66db\") " Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588714 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-credential-socket" pod "ce9d0b06-cc52-424e-85b4-9d53582d127c" (UID: "ce9d0b06-cc52-424e-85b4-9d53582d127c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.588717 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-data" pod "3fad2e2e-4765-463c-8062-fa51ddde66db" (UID: "3fad2e2e-4765-463c-8062-fa51ddde66db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589225 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-certs" pod "3fad2e2e-4765-463c-8062-fa51ddde66db" (UID: "3fad2e2e-4765-463c-8062-fa51ddde66db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.589652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589647 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.590512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589671 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.590512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589687 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.590512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589702 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.590512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589716 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fad2e2e-4765-463c-8062-fa51ddde66db-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.590512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589731 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.590512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589745 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.590512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589759 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce9d0b06-cc52-424e-85b4-9d53582d127c-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.590512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.589775 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.590512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.590258 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-envoy" pod "3fad2e2e-4765-463c-8062-fa51ddde66db" (UID: "3fad2e2e-4765-463c-8062-fa51ddde66db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.591009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.590532 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-kube-api-access-pzfxz" pod "ce9d0b06-cc52-424e-85b4-9d53582d127c" (UID: "ce9d0b06-cc52-424e-85b4-9d53582d127c"). InnerVolumeSpecName "kube-api-access-pzfxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:37.591009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.590685 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-podinfo" pod "ce9d0b06-cc52-424e-85b4-9d53582d127c" (UID: "ce9d0b06-cc52-424e-85b4-9d53582d127c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:37.591158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.591134 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-envoy" pod "ce9d0b06-cc52-424e-85b4-9d53582d127c" (UID: "ce9d0b06-cc52-424e-85b4-9d53582d127c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.591490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.591464 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-token" pod "ce9d0b06-cc52-424e-85b4-9d53582d127c" (UID: "ce9d0b06-cc52-424e-85b4-9d53582d127c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:37.591707 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.591681 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-token" pod "3fad2e2e-4765-463c-8062-fa51ddde66db" (UID: "3fad2e2e-4765-463c-8062-fa51ddde66db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:37.592313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.592284 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-kube-api-access-9bgbt" pod "3fad2e2e-4765-463c-8062-fa51ddde66db" (UID: "3fad2e2e-4765-463c-8062-fa51ddde66db"). InnerVolumeSpecName "kube-api-access-9bgbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:37.592601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.592582 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-podinfo" pod "3fad2e2e-4765-463c-8062-fa51ddde66db" (UID: "3fad2e2e-4765-463c-8062-fa51ddde66db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:37.609247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.609229 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:39:37.690896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.690866 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-kube-api-access-zbfn7\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-kube-api-access-zbfn7\") pod \"da392a7d-e24e-4dd1-b5f0-991320a94769\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " Nov 02 15:39:37.691089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.690909 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-kube-api-access-7sxjk\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-kube-api-access-7sxjk\") pod \"09f450e2-ade2-435a-87d4-fd0cf4693442\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " Nov 02 15:39:37.691089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.690937 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-token\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-token\") pod \"da392a7d-e24e-4dd1-b5f0-991320a94769\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " Nov 02 15:39:37.691089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691026 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-token\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-token\") pod \"09f450e2-ade2-435a-87d4-fd0cf4693442\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " Nov 02 15:39:37.691273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691120 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-credential-socket\") pod \"09f450e2-ade2-435a-87d4-fd0cf4693442\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " Nov 02 15:39:37.691273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691172 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-data\") pod \"da392a7d-e24e-4dd1-b5f0-991320a94769\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " Nov 02 15:39:37.691273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691210 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-podinfo\") pod \"da392a7d-e24e-4dd1-b5f0-991320a94769\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " Nov 02 15:39:37.691273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691246 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f450e2-ade2-435a-87d4-fd0cf4693442-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f450e2-ade2-435a-87d4-fd0cf4693442-istiod-ca-cert\") pod \"09f450e2-ade2-435a-87d4-fd0cf4693442\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " Nov 02 15:39:37.691512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691282 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-socket\") pod \"da392a7d-e24e-4dd1-b5f0-991320a94769\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " Nov 02 15:39:37.691512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691372 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-envoy\") pod \"09f450e2-ade2-435a-87d4-fd0cf4693442\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " Nov 02 15:39:37.691512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691403 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da392a7d-e24e-4dd1-b5f0-991320a94769-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da392a7d-e24e-4dd1-b5f0-991320a94769-istiod-ca-cert\") pod \"da392a7d-e24e-4dd1-b5f0-991320a94769\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " Nov 02 15:39:37.691512 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691464 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-podinfo\") pod \"09f450e2-ade2-435a-87d4-fd0cf4693442\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " Nov 02 15:39:37.691728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691512 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-socket\") pod \"09f450e2-ade2-435a-87d4-fd0cf4693442\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " Nov 02 15:39:37.691728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691560 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-certs\") pod \"09f450e2-ade2-435a-87d4-fd0cf4693442\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " Nov 02 15:39:37.691728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691620 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-certs\") pod \"da392a7d-e24e-4dd1-b5f0-991320a94769\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " Nov 02 15:39:37.691728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691647 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-envoy\") pod \"da392a7d-e24e-4dd1-b5f0-991320a94769\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " Nov 02 15:39:37.691728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691681 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-credential-socket\") pod \"da392a7d-e24e-4dd1-b5f0-991320a94769\" (UID: \"da392a7d-e24e-4dd1-b5f0-991320a94769\") " Nov 02 15:39:37.691728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691724 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-data\") pod \"09f450e2-ade2-435a-87d4-fd0cf4693442\" (UID: \"09f450e2-ade2-435a-87d4-fd0cf4693442\") " Nov 02 15:39:37.692078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.691874 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-credential-socket" pod "09f450e2-ade2-435a-87d4-fd0cf4693442" (UID: "09f450e2-ade2-435a-87d4-fd0cf4693442"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.692162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692071 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-data" pod "09f450e2-ade2-435a-87d4-fd0cf4693442" (UID: "09f450e2-ade2-435a-87d4-fd0cf4693442"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.692162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692089 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-data" pod "da392a7d-e24e-4dd1-b5f0-991320a94769" (UID: "da392a7d-e24e-4dd1-b5f0-991320a94769"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.692365 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692344 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da392a7d-e24e-4dd1-b5f0-991320a94769-istiod-ca-cert" pod "da392a7d-e24e-4dd1-b5f0-991320a94769" (UID: "da392a7d-e24e-4dd1-b5f0-991320a94769"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:37.692604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692572 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.692682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692611 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.692682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692627 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzfxz\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-kube-api-access-pzfxz\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.692682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692643 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgbt\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-kube-api-access-9bgbt\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.692682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692658 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.692682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692672 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.693099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692686 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.693099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692699 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.693099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692712 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.693099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692724 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fad2e2e-4765-463c-8062-fa51ddde66db-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.693099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692737 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce9d0b06-cc52-424e-85b4-9d53582d127c-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.693099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.692750 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.693099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.693072 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f450e2-ade2-435a-87d4-fd0cf4693442-istiod-ca-cert" pod "09f450e2-ade2-435a-87d4-fd0cf4693442" (UID: "09f450e2-ade2-435a-87d4-fd0cf4693442"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:37.693484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.693123 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-socket" pod "da392a7d-e24e-4dd1-b5f0-991320a94769" (UID: "da392a7d-e24e-4dd1-b5f0-991320a94769"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.693546 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.693512 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-certs" pod "09f450e2-ade2-435a-87d4-fd0cf4693442" (UID: "09f450e2-ade2-435a-87d4-fd0cf4693442"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.693829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.693785 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-credential-socket" pod "da392a7d-e24e-4dd1-b5f0-991320a94769" (UID: "da392a7d-e24e-4dd1-b5f0-991320a94769"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.693829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.693819 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-socket" pod "09f450e2-ade2-435a-87d4-fd0cf4693442" (UID: "09f450e2-ade2-435a-87d4-fd0cf4693442"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.694009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.693865 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-certs" pod "da392a7d-e24e-4dd1-b5f0-991320a94769" (UID: "da392a7d-e24e-4dd1-b5f0-991320a94769"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.694841 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.694808 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-token" pod "09f450e2-ade2-435a-87d4-fd0cf4693442" (UID: "09f450e2-ade2-435a-87d4-fd0cf4693442"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:37.695019 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.694995 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-podinfo" pod "09f450e2-ade2-435a-87d4-fd0cf4693442" (UID: "09f450e2-ade2-435a-87d4-fd0cf4693442"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:37.695805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.695787 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-envoy" pod "da392a7d-e24e-4dd1-b5f0-991320a94769" (UID: "da392a7d-e24e-4dd1-b5f0-991320a94769"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.696144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.696119 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-kube-api-access-7sxjk" pod "09f450e2-ade2-435a-87d4-fd0cf4693442" (UID: "09f450e2-ade2-435a-87d4-fd0cf4693442"). InnerVolumeSpecName "kube-api-access-7sxjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:37.696259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.696239 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-envoy" pod "09f450e2-ade2-435a-87d4-fd0cf4693442" (UID: "09f450e2-ade2-435a-87d4-fd0cf4693442"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:37.696375 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.696355 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-podinfo" pod "da392a7d-e24e-4dd1-b5f0-991320a94769" (UID: "da392a7d-e24e-4dd1-b5f0-991320a94769"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:37.696966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.696944 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-token" pod "da392a7d-e24e-4dd1-b5f0-991320a94769" (UID: "da392a7d-e24e-4dd1-b5f0-991320a94769"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:37.697440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.697415 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-kube-api-access-zbfn7" pod "da392a7d-e24e-4dd1-b5f0-991320a94769" (UID: "da392a7d-e24e-4dd1-b5f0-991320a94769"). InnerVolumeSpecName "kube-api-access-zbfn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:37.794862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794781 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.794862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794823 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.794862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794838 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f450e2-ade2-435a-87d4-fd0cf4693442-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.794862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794851 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.794862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794864 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794873 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da392a7d-e24e-4dd1-b5f0-991320a94769-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794885 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f450e2-ade2-435a-87d4-fd0cf4693442-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794899 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794912 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f450e2-ade2-435a-87d4-fd0cf4693442-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794924 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794933 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794944 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da392a7d-e24e-4dd1-b5f0-991320a94769-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794952 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbfn7\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-kube-api-access-zbfn7\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.794968 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sxjk\" (UniqueName: \"kubernetes.io/projected/09f450e2-ade2-435a-87d4-fd0cf4693442-kube-api-access-7sxjk\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:37.795163 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:37.795000 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da392a7d-e24e-4dd1-b5f0-991320a94769-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:38.346723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.346696 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" Nov 02 15:39:38.346893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.346733 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb" event={"ID":"3fad2e2e-4765-463c-8062-fa51ddde66db","Type":"ContainerDied","Data":"a32ee5c9e55e9cf65a0937c4bb4fed3d5a69d6c0ea0f68f266b55179729c962a"} Nov 02 15:39:38.346893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.346810 5433 scope.go:122] "RemoveContainer" containerID="a49d3d2105fb431182dcb8c735e13c8f2c62070910b23a20e20511834a85a622" Nov 02 15:39:38.348462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.348439 5433 generic.go:356] "Generic (PLEG): container finished" podID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerID="f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c" exitCode=0 Nov 02 15:39:38.348544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.348525 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" Nov 02 15:39:38.348544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.348530 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" event={"ID":"09f450e2-ade2-435a-87d4-fd0cf4693442","Type":"ContainerDied","Data":"f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c"} Nov 02 15:39:38.348630 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.348574 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5" event={"ID":"09f450e2-ade2-435a-87d4-fd0cf4693442","Type":"ContainerDied","Data":"744d0cc9de547b5e8787c734a5a359c295fb76f97e8ae35dd5a0b1c05bfbd386"} Nov 02 15:39:38.350278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.350257 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" Nov 02 15:39:38.350419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.350256 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv" event={"ID":"ce9d0b06-cc52-424e-85b4-9d53582d127c","Type":"ContainerDied","Data":"c7346fba3787e456b81d0775a87e9177e7b109b52144fd364a62ea6dec6cc59e"} Nov 02 15:39:38.351886 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.351859 5433 generic.go:356] "Generic (PLEG): container finished" podID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerID="9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9" exitCode=0 Nov 02 15:39:38.352028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.351914 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" Nov 02 15:39:38.352028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.351932 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" event={"ID":"da392a7d-e24e-4dd1-b5f0-991320a94769","Type":"ContainerDied","Data":"9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9"} Nov 02 15:39:38.352028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.351970 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb" event={"ID":"da392a7d-e24e-4dd1-b5f0-991320a94769","Type":"ContainerDied","Data":"29ecb082e4f843b0690168947a9a7b2afb6c5f42fe049a3c37833f647fd55fb6"} Nov 02 15:39:38.359617 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.359575 5433 scope.go:122] "RemoveContainer" containerID="f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c" Nov 02 15:39:38.369816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.369798 5433 scope.go:122] "RemoveContainer" containerID="f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c" Nov 02 15:39:38.370156 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:38.370134 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c\": container with ID starting with f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c not found: ID does not exist" containerID="f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c" Nov 02 15:39:38.370260 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.370167 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c"} err="failed to get container status \"f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c\": rpc error: code = NotFound desc = could not find container \"f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c\": container with ID starting with f4b36bfb68e18f7e272f6ac8ad7f0357bc19cf835fb23001c82a17fe37163f4c not found: ID does not exist" Nov 02 15:39:38.370260 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.370185 5433 scope.go:122] "RemoveContainer" containerID="d3f205f7d8d9b5ff08a512640270d7ab76d7f4a4755fb8aab492065c1eab4478" Nov 02 15:39:38.379149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.379132 5433 scope.go:122] "RemoveContainer" containerID="9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9" Nov 02 15:39:38.388195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.388168 5433 scope.go:122] "RemoveContainer" containerID="9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9" Nov 02 15:39:38.388466 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:38.388436 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9\": container with ID starting with 9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9 not found: ID does not exist" containerID="9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9" Nov 02 15:39:38.388523 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.388477 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9"} err="failed to get container status \"9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9\": rpc error: code = NotFound desc = could not find container \"9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9\": container with ID starting with 9fa3b3ddf9085f99c482383816a39600f1719908ed7182646642245fb78510f9 not found: ID does not exist" Nov 02 15:39:38.503015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.502903 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb"] Nov 02 15:39:38.549565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.549527 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6hlvb"] Nov 02 15:39:38.611394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.611347 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv"] Nov 02 15:39:38.616908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.616870 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jhvv"] Nov 02 15:39:38.728894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.728861 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb"] Nov 02 15:39:38.733440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.733412 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s6jxb"] Nov 02 15:39:38.836044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.835953 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5"] Nov 02 15:39:38.840050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:38.840019 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4d7h5"] Nov 02 15:39:39.553799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.553734 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:39:39.723158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723127 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-envoy\") pod \"2696aeeb-a326-4622-aafd-c7665e1269cc\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " Nov 02 15:39:39.723352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723175 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-token\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-token\") pod \"2696aeeb-a326-4622-aafd-c7665e1269cc\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " Nov 02 15:39:39.723352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723274 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-socket\") pod \"2696aeeb-a326-4622-aafd-c7665e1269cc\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " Nov 02 15:39:39.723352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723304 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-podinfo\") pod \"2696aeeb-a326-4622-aafd-c7665e1269cc\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " Nov 02 15:39:39.723528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723364 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-kube-api-access-7j7zx\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-kube-api-access-7j7zx\") pod \"2696aeeb-a326-4622-aafd-c7665e1269cc\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " Nov 02 15:39:39.723528 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723426 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-data\") pod \"2696aeeb-a326-4622-aafd-c7665e1269cc\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " Nov 02 15:39:39.723641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723559 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-certs\") pod \"2696aeeb-a326-4622-aafd-c7665e1269cc\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " Nov 02 15:39:39.723641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723582 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-socket" pod "2696aeeb-a326-4622-aafd-c7665e1269cc" (UID: "2696aeeb-a326-4622-aafd-c7665e1269cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.723641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723605 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2696aeeb-a326-4622-aafd-c7665e1269cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2696aeeb-a326-4622-aafd-c7665e1269cc-istiod-ca-cert\") pod \"2696aeeb-a326-4622-aafd-c7665e1269cc\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " Nov 02 15:39:39.723802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723664 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-credential-socket\") pod \"2696aeeb-a326-4622-aafd-c7665e1269cc\" (UID: \"2696aeeb-a326-4622-aafd-c7665e1269cc\") " Nov 02 15:39:39.723802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723758 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-data" pod "2696aeeb-a326-4622-aafd-c7665e1269cc" (UID: "2696aeeb-a326-4622-aafd-c7665e1269cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.723900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.723807 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-certs" pod "2696aeeb-a326-4622-aafd-c7665e1269cc" (UID: "2696aeeb-a326-4622-aafd-c7665e1269cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.724067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.724042 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-credential-socket" pod "2696aeeb-a326-4622-aafd-c7665e1269cc" (UID: "2696aeeb-a326-4622-aafd-c7665e1269cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.724162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.724059 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2696aeeb-a326-4622-aafd-c7665e1269cc-istiod-ca-cert" pod "2696aeeb-a326-4622-aafd-c7665e1269cc" (UID: "2696aeeb-a326-4622-aafd-c7665e1269cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:39.724227 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.724218 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.724283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.724235 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.724283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.724252 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2696aeeb-a326-4622-aafd-c7665e1269cc-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.724283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.724266 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.724283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.724280 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.725535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.725497 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-podinfo" pod "2696aeeb-a326-4622-aafd-c7665e1269cc" (UID: "2696aeeb-a326-4622-aafd-c7665e1269cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:39.725646 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.725629 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-token" pod "2696aeeb-a326-4622-aafd-c7665e1269cc" (UID: "2696aeeb-a326-4622-aafd-c7665e1269cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:39.725904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.725875 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-envoy" pod "2696aeeb-a326-4622-aafd-c7665e1269cc" (UID: "2696aeeb-a326-4622-aafd-c7665e1269cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.726296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.726275 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-kube-api-access-7j7zx" pod "2696aeeb-a326-4622-aafd-c7665e1269cc" (UID: "2696aeeb-a326-4622-aafd-c7665e1269cc"). InnerVolumeSpecName "kube-api-access-7j7zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:39.825602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.825529 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.825602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.825558 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.825602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.825569 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2696aeeb-a326-4622-aafd-c7665e1269cc-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.825602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:39.825578 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j7zx\" (UniqueName: \"kubernetes.io/projected/2696aeeb-a326-4622-aafd-c7665e1269cc-kube-api-access-7j7zx\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:40.371157 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.371124 5433 generic.go:356] "Generic (PLEG): container finished" podID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerID="dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f" exitCode=0 Nov 02 15:39:40.371374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.371178 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" Nov 02 15:39:40.371374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.371220 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" event={"ID":"2696aeeb-a326-4622-aafd-c7665e1269cc","Type":"ContainerDied","Data":"dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f"} Nov 02 15:39:40.371374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.371280 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8" event={"ID":"2696aeeb-a326-4622-aafd-c7665e1269cc","Type":"ContainerDied","Data":"9069500f06f62e48a8f2733ead06628edea37007982bc2d29de66b9e192817c6"} Nov 02 15:39:40.371374 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.371314 5433 scope.go:122] "RemoveContainer" containerID="dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f" Nov 02 15:39:40.398338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.384540 5433 scope.go:122] "RemoveContainer" containerID="fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20" Nov 02 15:39:40.402023 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.401504 5433 scope.go:122] "RemoveContainer" containerID="dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f" Nov 02 15:39:40.402023 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:40.401926 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f\": container with ID starting with dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f not found: ID does not exist" containerID="dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f" Nov 02 15:39:40.402131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.401968 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f"} err="failed to get container status \"dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f\": rpc error: code = NotFound desc = could not find container \"dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f\": container with ID starting with dfbbd0e9f92c7bf9c70f7bb267ce3de911ac14c587db0f0a475e7348df6d9d1f not found: ID does not exist" Nov 02 15:39:40.402131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.402037 5433 scope.go:122] "RemoveContainer" containerID="fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20" Nov 02 15:39:40.402388 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:40.402363 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20\": container with ID starting with fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20 not found: ID does not exist" containerID="fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20" Nov 02 15:39:40.402457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.402397 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20"} err="failed to get container status \"fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20\": rpc error: code = NotFound desc = could not find container \"fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20\": container with ID starting with fbfbac18d62ff58dededd8617e1e803d04a798a74d7c6f4734f62b7b3688fc20 not found: ID does not exist" Nov 02 15:39:40.463631 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.463598 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" path="/var/lib/kubelet/pods/09f450e2-ade2-435a-87d4-fd0cf4693442/volumes" Nov 02 15:39:40.464095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.464080 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" path="/var/lib/kubelet/pods/3fad2e2e-4765-463c-8062-fa51ddde66db/volumes" Nov 02 15:39:40.464495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.464484 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" path="/var/lib/kubelet/pods/ce9d0b06-cc52-424e-85b4-9d53582d127c/volumes" Nov 02 15:39:40.465203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.465189 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" path="/var/lib/kubelet/pods/da392a7d-e24e-4dd1-b5f0-991320a94769/volumes" Nov 02 15:39:40.574595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.574564 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8"] Nov 02 15:39:40.619345 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:40.619304 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-qptx8"] Nov 02 15:39:41.013029 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:39:41.012990 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:39:41.493009 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:41.492966 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:42.457722 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:42.457688 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" path="/var/lib/kubelet/pods/2696aeeb-a326-4622-aafd-c7665e1269cc/volumes" Nov 02 15:39:42.731943 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:42.731864 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:46.588388 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:46.588356 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:47.231607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.231570 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj"] Nov 02 15:39:47.232237 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232221 5433 cpu_manager.go:414] "containerMap: removing container" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" Nov 02 15:39:47.232284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232238 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" Nov 02 15:39:47.232284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232251 5433 cpu_manager.go:414] "containerMap: removing container" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" Nov 02 15:39:47.232284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232256 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" Nov 02 15:39:47.232284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232266 5433 cpu_manager.go:414] "containerMap: removing container" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" Nov 02 15:39:47.232284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232271 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" Nov 02 15:39:47.232284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232278 5433 cpu_manager.go:414] "containerMap: removing container" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" Nov 02 15:39:47.232284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232283 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232294 5433 cpu_manager.go:414] "containerMap: removing container" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232298 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232313 5433 cpu_manager.go:414] "containerMap: removing container" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232317 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232397 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fad2e2e-4765-463c-8062-fa51ddde66db" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232404 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232413 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9d0b06-cc52-424e-85b4-9d53582d127c" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232420 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="da392a7d-e24e-4dd1-b5f0-991320a94769" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232427 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2696aeeb-a326-4622-aafd-c7665e1269cc" containerName="istio-proxy" Nov 02 15:39:47.232497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.232433 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f450e2-ade2-435a-87d4-fd0cf4693442" containerName="istio-proxy" Nov 02 15:39:47.236179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.236158 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.278121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.278089 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5"] Nov 02 15:39:47.282398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.282376 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75"] Nov 02 15:39:47.282559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.282531 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.287203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.287181 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.294226 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.294205 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9"] Nov 02 15:39:47.298698 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.298678 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.317156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317134 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3940e38e-34c7-49dd-b315-8a03373d0f59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.317271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317164 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.317271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317189 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.317346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317272 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.317346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317324 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.317406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317344 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.317437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317400 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.317472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317440 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.317503 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317496 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.317556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317544 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.317594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317572 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e5919a-b07b-4163-b8fa-910fd2352f06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.317594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317591 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.317660 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317610 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.317660 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317641 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.317719 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317687 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kw2t\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-kube-api-access-4kw2t\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.317794 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317723 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.317794 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317747 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptpjj\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-kube-api-access-ptpjj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.317794 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317779 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.317882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317801 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54fee09-3c5a-4601-837d-5ca649fdbee9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.317882 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317837 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.317944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317898 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.317944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317914 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.318015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317945 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.318015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.317999 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khbxq\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-kube-api-access-khbxq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.318084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.318022 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.318084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.318061 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.318142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.318083 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.321510 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.321493 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x6s8k\"" Nov 02 15:39:47.331480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.331456 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj"] Nov 02 15:39:47.385626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.385505 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75"] Nov 02 15:39:47.395023 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.392731 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5"] Nov 02 15:39:47.403177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.403135 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9"] Nov 02 15:39:47.418865 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.418827 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.419038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.418868 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khbxq\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-kube-api-access-khbxq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.419038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.418895 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.419038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.418929 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.419038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.418957 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.419038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419005 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3940e38e-34c7-49dd-b315-8a03373d0f59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.419351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419038 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d72143-9cba-4630-97c1-094c28c4e01c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.419351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419065 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.419351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419096 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.419351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419123 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.419351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419150 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.419351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419200 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.419351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419246 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.419729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419611 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.419729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.419642 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.420034 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420005 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.420174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420111 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.420174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420158 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.420285 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420195 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.420285 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420251 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.420393 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420289 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e5919a-b07b-4163-b8fa-910fd2352f06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.420393 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420317 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.420393 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420346 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.420393 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420362 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.420393 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420371 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.420650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420421 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kw2t\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-kube-api-access-4kw2t\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.420650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420457 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.420650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420492 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.420650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420499 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.420650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420540 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptpjj\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-kube-api-access-ptpjj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.420650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420544 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3940e38e-34c7-49dd-b315-8a03373d0f59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.420650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420565 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.420650 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.420636 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.421187 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421153 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.421366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421344 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e5919a-b07b-4163-b8fa-910fd2352f06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.421621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421597 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.421724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421704 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.421788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421728 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.421788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421748 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54fee09-3c5a-4601-837d-5ca649fdbee9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.421888 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421794 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.421888 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421832 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d72143-9cba-4630-97c1-094c28c4e01c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.421888 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421867 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zxgd\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-kube-api-access-8zxgd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.422076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421899 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.422076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421925 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.422076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.421969 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.422076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.422023 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.424497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.424328 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.424497 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.424335 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.424701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.424595 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.424757 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.424693 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.424808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.424781 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54fee09-3c5a-4601-837d-5ca649fdbee9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.425004 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.424940 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.425940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.425914 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.427788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.427759 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.428261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.428241 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.428261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.428251 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.429679 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.429658 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.479460 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.479427 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.479625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.479518 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.479735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.479716 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khbxq\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-kube-api-access-khbxq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.479795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.479736 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptpjj\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-kube-api-access-ptpjj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2vz75\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.480235 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.480212 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kw2t\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-kube-api-access-4kw2t\") pod \"maas-default-gateway-openshift-default-7f5667f86c-qgfx5\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.486148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.486095 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x2fcj\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.523092 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523051 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523100 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523188 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d72143-9cba-4630-97c1-094c28c4e01c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523218 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523262 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523244 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523438 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523481 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523561 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523484 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523598 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d72143-9cba-4630-97c1-094c28c4e01c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523639 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zxgd\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-kube-api-access-8zxgd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523841 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523814 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523856 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.523923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.523875 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.524114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.524096 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d72143-9cba-4630-97c1-094c28c4e01c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.525737 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.525718 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.526065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.526043 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d72143-9cba-4630-97c1-094c28c4e01c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.550780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.550755 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:47.580724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.580695 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.580910 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.580889 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zxgd\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-kube-api-access-8zxgd\") pod \"maas-default-gateway-openshift-default-7f5667f86c-92bt9\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.598844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.598811 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:47.605470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.605247 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:47.614536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.614365 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:47.927296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.927257 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj"] Nov 02 15:39:47.928204 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:39:47.928174 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e5919a_b07b_4163_b8fa_910fd2352f06.slice/crio-89687ab38cb93cb8bc2703e4f6ff1de22374b75c1b06d82d2b7c7f41675d5a69 WatchSource:0}: Error finding container 89687ab38cb93cb8bc2703e4f6ff1de22374b75c1b06d82d2b7c7f41675d5a69: Status 404 returned error can't find the container with id 89687ab38cb93cb8bc2703e4f6ff1de22374b75c1b06d82d2b7c7f41675d5a69 Nov 02 15:39:47.930842 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.930807 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:47.930923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.930882 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:47.930923 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.930916 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:47.999499 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:47.999471 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5"] Nov 02 15:39:48.000490 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:39:48.000469 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54fee09_3c5a_4601_837d_5ca649fdbee9.slice/crio-ae520c517e7eb1ad88bf2004e1daa8d40dea6f4f41d14d9e7c7e1e9e4ce60baa WatchSource:0}: Error finding container ae520c517e7eb1ad88bf2004e1daa8d40dea6f4f41d14d9e7c7e1e9e4ce60baa: Status 404 returned error can't find the container with id ae520c517e7eb1ad88bf2004e1daa8d40dea6f4f41d14d9e7c7e1e9e4ce60baa Nov 02 15:39:48.003111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.003083 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:48.003183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.003147 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:48.003183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.003174 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:48.041417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.041383 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9"] Nov 02 15:39:48.042812 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:39:48.042785 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d72143_9cba_4630_97c1_094c28c4e01c.slice/crio-da8acb53a91208ad8dc7e5b0103e55fd9c3190e9e863c18cea02a7db3bdea9b0 WatchSource:0}: Error finding container da8acb53a91208ad8dc7e5b0103e55fd9c3190e9e863c18cea02a7db3bdea9b0: Status 404 returned error can't find the container with id da8acb53a91208ad8dc7e5b0103e55fd9c3190e9e863c18cea02a7db3bdea9b0 Nov 02 15:39:48.047223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.047181 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:48.047333 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.047281 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:48.047404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.047333 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:48.050437 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.050362 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75"] Nov 02 15:39:48.059094 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:39:48.059067 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3940e38e_34c7_49dd_b315_8a03373d0f59.slice/crio-16c821e866c35d15acbbb21d2df7de1c1e182047abb0d58c6e42105aaf1b1558 WatchSource:0}: Error finding container 16c821e866c35d15acbbb21d2df7de1c1e182047abb0d58c6e42105aaf1b1558: Status 404 returned error can't find the container with id 16c821e866c35d15acbbb21d2df7de1c1e182047abb0d58c6e42105aaf1b1558 Nov 02 15:39:48.062228 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.062197 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:48.062415 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.062260 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:48.062415 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.062290 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:48.444818 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.444763 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" event={"ID":"3940e38e-34c7-49dd-b315-8a03373d0f59","Type":"ContainerStarted","Data":"59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d"} Nov 02 15:39:48.445189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.445163 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" event={"ID":"3940e38e-34c7-49dd-b315-8a03373d0f59","Type":"ContainerStarted","Data":"16c821e866c35d15acbbb21d2df7de1c1e182047abb0d58c6e42105aaf1b1558"} Nov 02 15:39:48.448018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.447954 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" event={"ID":"d54fee09-3c5a-4601-837d-5ca649fdbee9","Type":"ContainerStarted","Data":"8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36"} Nov 02 15:39:48.448018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.448017 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" event={"ID":"d54fee09-3c5a-4601-837d-5ca649fdbee9","Type":"ContainerStarted","Data":"ae520c517e7eb1ad88bf2004e1daa8d40dea6f4f41d14d9e7c7e1e9e4ce60baa"} Nov 02 15:39:48.459244 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.459214 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" event={"ID":"64d72143-9cba-4630-97c1-094c28c4e01c","Type":"ContainerStarted","Data":"8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf"} Nov 02 15:39:48.459385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.459271 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" event={"ID":"64d72143-9cba-4630-97c1-094c28c4e01c","Type":"ContainerStarted","Data":"da8acb53a91208ad8dc7e5b0103e55fd9c3190e9e863c18cea02a7db3bdea9b0"} Nov 02 15:39:48.459385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.459295 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" event={"ID":"48e5919a-b07b-4163-b8fa-910fd2352f06","Type":"ContainerStarted","Data":"814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164"} Nov 02 15:39:48.459385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.459320 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" event={"ID":"48e5919a-b07b-4163-b8fa-910fd2352f06","Type":"ContainerStarted","Data":"89687ab38cb93cb8bc2703e4f6ff1de22374b75c1b06d82d2b7c7f41675d5a69"} Nov 02 15:39:48.553818 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.553784 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:48.554024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.553929 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 15:39:48.554749 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.554710 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 15:39:48.565014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.564783 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podStartSLOduration=1.564764337 podStartE2EDuration="1.564764337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:48.558302672 +0000 UTC m=+2492.231505867" watchObservedRunningTime="2026-11-02 15:39:48.564764337 +0000 UTC m=+2492.237967534" Nov 02 15:39:48.599014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.598963 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:48.605454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.605423 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:48.614968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.614944 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:48.673000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.672918 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podStartSLOduration=1.672902227 podStartE2EDuration="1.672902227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:48.670767295 +0000 UTC m=+2492.343970493" watchObservedRunningTime="2026-11-02 15:39:48.672902227 +0000 UTC m=+2492.346105496" Nov 02 15:39:48.788652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.786186 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podStartSLOduration=1.786166656 podStartE2EDuration="1.786166656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:48.78446127 +0000 UTC m=+2492.457664469" watchObservedRunningTime="2026-11-02 15:39:48.786166656 +0000 UTC m=+2492.459369852" Nov 02 15:39:48.899122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:48.899050 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podStartSLOduration=1.89903229 podStartE2EDuration="1.89903229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:48.897158946 +0000 UTC m=+2492.570362131" watchObservedRunningTime="2026-11-02 15:39:48.89903229 +0000 UTC m=+2492.572235489" Nov 02 15:39:49.662999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:49.590015 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 15:39:49.662999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:49.590077 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 15:39:49.733239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:49.699132 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-qgfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 15:39:49.733239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:49.699202 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 15:39:49.733789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:49.733757 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2vz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 15:39:49.734123 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:49.734095 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 15:39:49.734327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:49.734030 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-92bt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 15:39:49.759131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:49.759080 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 15:39:50.085200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.085077 5433 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:50.557001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.556566 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 15:39:50.557225 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.556655 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 15:39:50.591115 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.591056 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 15:39:50.602451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.602413 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-qgfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 15:39:50.602754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.602713 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 15:39:50.610425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.610394 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2vz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 15:39:50.610676 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.610638 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 15:39:50.617313 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.617283 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-92bt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 15:39:50.617515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:50.617490 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 15:39:51.441999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.439546 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:51.564312 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.564277 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 15:39:51.564596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.564562 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 15:39:51.607014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.600436 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-qgfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 15:39:51.607014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.600497 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 15:39:51.622665 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.622628 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2vz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 15:39:51.622958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.622928 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 15:39:51.623446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.623421 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-92bt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 15:39:51.623602 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.623582 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 15:39:51.713178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.713064 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:51.741120 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.741068 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:51.819006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:51.818529 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:52.454281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.451078 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:52.455351 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.455316 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:39:52.552470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.552432 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 15:39:52.552748 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.552719 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 15:39:52.605000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.562324 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:52.605000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.601381 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-qgfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 15:39:52.605000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.601445 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 15:39:52.725068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.717186 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-92bt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 15:39:52.738003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.729062 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 15:39:52.738298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.732877 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2vz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 15:39:52.738451 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.738419 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 15:39:52.821037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.816121 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:52.931003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:52.922530 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:39:53.034006 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.033893 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:53.446323 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.446279 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:53.456601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.456566 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 15:39:53.551442 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.551402 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 15:39:53.552468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.552433 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 15:39:53.613847 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.613805 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-qgfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 15:39:53.614149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.614103 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 15:39:53.614462 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.614443 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2vz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 15:39:53.614618 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.614598 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 15:39:53.615124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.615051 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-92bt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 15:39:53.615293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.615274 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 15:39:53.770963 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.768619 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:53.884905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:53.884089 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:54.215035 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.170158 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:54.276024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.272933 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:39:54.564056 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.561853 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 15:39:54.564056 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.561915 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 15:39:54.569404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.569323 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:54.626031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.602338 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-qgfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 15:39:54.626031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.602407 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 15:39:54.626031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.611099 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2vz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 15:39:54.626031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.611174 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 15:39:54.626031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.619839 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-92bt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 15:39:54.626031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.619890 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 15:39:54.711314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.711233 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:54.809152 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.809096 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:54.811126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.811104 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:39:54.903005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:54.900340 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:55.273637 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.273529 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:55.494435 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.494402 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:39:55.502921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.502885 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:39:55.555608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.555519 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 15:39:55.555866 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.555834 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 15:39:55.600406 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.600369 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-qgfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 15:39:55.600795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.600435 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 15:39:55.606596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.606562 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2vz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 15:39:55.606738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.606620 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 15:39:55.622165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.622131 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-92bt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 15:39:55.622473 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:55.622418 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 15:39:56.551636 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:56.551596 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 15:39:56.551861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:56.551670 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 15:39:56.600180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:56.600139 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-qgfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 15:39:56.600395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:56.600206 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 15:39:56.606599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:56.606558 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2vz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 15:39:56.607143 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:56.606639 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 15:39:56.616044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:56.616007 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-92bt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 15:39:56.616215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:56.616065 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 15:39:57.609001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:57.606423 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2vz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 15:39:57.609001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:57.606492 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 15:39:58.600644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.599140 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:39:58.600644 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.599206 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:58.616912 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.616874 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-92bt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:39:58.617417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.616955 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:58.641185 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.641139 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-qgfx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:39:58.641347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.641209 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:58.673710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.673671 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:58.674577 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.674557 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:58.676823 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.676807 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:39:58.746894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.746372 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:58.802751 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.802702 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:58.803866 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.803833 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:58.804900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.804882 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:39:58.829390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.829357 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:58.829699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.829678 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:58.830766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.830748 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:39:58.921652 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:58.921615 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:39:59.067801 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:59.067769 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:59.069020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:59.068930 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:59.069754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:59.069729 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:39:59.457103 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:39:59.457069 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:00.297038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:00.295932 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:00.579735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:00.579645 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:01.813589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:01.813551 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:02.018993 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.018878 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:02.196776 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.196713 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:02.390947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.390912 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw"] Nov 02 15:40:02.396539 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.396513 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:02.445295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.445257 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e3b351\"" Nov 02 15:40:02.491663 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.491576 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e3b351-dockercfg-tf45c\"" Nov 02 15:40:02.494790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.494761 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw"] Nov 02 15:40:02.516672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.516641 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw\" (UID: \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:02.516849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.516679 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ntj\" (UniqueName: \"kubernetes.io/projected/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-kube-api-access-m2ntj\") pod \"payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw\" (UID: \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:02.618199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.618164 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw\" (UID: \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:02.618199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.618200 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ntj\" (UniqueName: \"kubernetes.io/projected/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-kube-api-access-m2ntj\") pod \"payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw\" (UID: \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:02.618876 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.618855 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw\" (UID: \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:02.676536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.676505 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ntj\" (UniqueName: \"kubernetes.io/projected/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-kube-api-access-m2ntj\") pod \"payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw\" (UID: \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:02.712755 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:02.712717 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:03.084183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:03.084153 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw"] Nov 02 15:40:03.086097 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:40:03.086054 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea5c62e_aa29_4f83_8ca9_abb02750fcfa.slice/crio-0fa8d7d7b5cbfe3200b55305370c29e9ca6083d0cab3196c1c685ae2cf3e7257 WatchSource:0}: Error finding container 0fa8d7d7b5cbfe3200b55305370c29e9ca6083d0cab3196c1c685ae2cf3e7257: Status 404 returned error can't find the container with id 0fa8d7d7b5cbfe3200b55305370c29e9ca6083d0cab3196c1c685ae2cf3e7257 Nov 02 15:40:03.221899 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:03.221863 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" event={"ID":"bea5c62e-aa29-4f83-8ca9-abb02750fcfa","Type":"ContainerStarted","Data":"9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17"} Nov 02 15:40:03.222134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:03.221916 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" event={"ID":"bea5c62e-aa29-4f83-8ca9-abb02750fcfa","Type":"ContainerStarted","Data":"0fa8d7d7b5cbfe3200b55305370c29e9ca6083d0cab3196c1c685ae2cf3e7257"} Nov 02 15:40:03.222134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:03.222117 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:03.335855 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:03.335744 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" podStartSLOduration=1.335726106 podStartE2EDuration="1.335726106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:03.334084603 +0000 UTC m=+2507.007287799" watchObservedRunningTime="2026-11-02 15:40:03.335726106 +0000 UTC m=+2507.008929300" Nov 02 15:40:05.109837 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:05.109806 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:10.595790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:10.595756 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:11.012417 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:40:11.012377 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:40:14.232214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:14.232182 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:40:15.293373 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.293341 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b"] Nov 02 15:40:15.293890 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.293585 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" podUID="d0638183-955f-4411-8f7b-49c24124e8a6" containerName="maas-api" containerID="cri-o://2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9" gracePeriod=30 Nov 02 15:40:15.393395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.393357 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:15.625795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.625760 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:40:15.763999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.763497 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-metrics-certs\") pod \"d0638183-955f-4411-8f7b-49c24124e8a6\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " Nov 02 15:40:15.763999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.763649 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0638183-955f-4411-8f7b-49c24124e8a6-kube-api-access-zgbq7\" (UniqueName: \"kubernetes.io/projected/d0638183-955f-4411-8f7b-49c24124e8a6-kube-api-access-zgbq7\") pod \"d0638183-955f-4411-8f7b-49c24124e8a6\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " Nov 02 15:40:15.763999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.763772 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls\") pod \"d0638183-955f-4411-8f7b-49c24124e8a6\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " Nov 02 15:40:15.763999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.763893 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0638183-955f-4411-8f7b-49c24124e8a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0638183-955f-4411-8f7b-49c24124e8a6-openshift-service-ca\") pod \"d0638183-955f-4411-8f7b-49c24124e8a6\" (UID: \"d0638183-955f-4411-8f7b-49c24124e8a6\") " Nov 02 15:40:15.774256 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.771648 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0638183-955f-4411-8f7b-49c24124e8a6-openshift-service-ca" pod "d0638183-955f-4411-8f7b-49c24124e8a6" (UID: "d0638183-955f-4411-8f7b-49c24124e8a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:40:15.783082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.783035 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls" pod "d0638183-955f-4411-8f7b-49c24124e8a6" (UID: "d0638183-955f-4411-8f7b-49c24124e8a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:40:15.783538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.783509 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0638183-955f-4411-8f7b-49c24124e8a6-kube-api-access-zgbq7" pod "d0638183-955f-4411-8f7b-49c24124e8a6" (UID: "d0638183-955f-4411-8f7b-49c24124e8a6"). InnerVolumeSpecName "kube-api-access-zgbq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:15.785670 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.785640 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-metrics-certs" pod "d0638183-955f-4411-8f7b-49c24124e8a6" (UID: "d0638183-955f-4411-8f7b-49c24124e8a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:40:15.865763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.865685 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0638183-955f-4411-8f7b-49c24124e8a6-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:15.865763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.865713 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:15.865763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.865723 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbq7\" (UniqueName: \"kubernetes.io/projected/d0638183-955f-4411-8f7b-49c24124e8a6-kube-api-access-zgbq7\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:15.865763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:15.865731 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0638183-955f-4411-8f7b-49c24124e8a6-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:16.305787 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:16.305754 5433 generic.go:356] "Generic (PLEG): container finished" podID="d0638183-955f-4411-8f7b-49c24124e8a6" containerID="2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9" exitCode=0 Nov 02 15:40:16.306305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:16.305800 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" event={"ID":"d0638183-955f-4411-8f7b-49c24124e8a6","Type":"ContainerDied","Data":"2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9"} Nov 02 15:40:16.306305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:16.305806 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" Nov 02 15:40:16.306305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:16.305839 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b" event={"ID":"d0638183-955f-4411-8f7b-49c24124e8a6","Type":"ContainerDied","Data":"6ff44db5e6c7b5885a72d277324a9458f6e1aafaa9abb233c4b779d7f263a929"} Nov 02 15:40:16.306305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:16.305859 5433 scope.go:122] "RemoveContainer" containerID="2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9" Nov 02 15:40:16.318287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:16.318264 5433 scope.go:122] "RemoveContainer" containerID="2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9" Nov 02 15:40:16.318553 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:40:16.318531 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9\": container with ID starting with 2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9 not found: ID does not exist" containerID="2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9" Nov 02 15:40:16.318645 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:16.318559 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9"} err="failed to get container status \"2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9\": rpc error: code = NotFound desc = could not find container \"2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9\": container with ID starting with 2424d0ca17d96c7df1b604aabfb0479fcf99af4a63264e1e1facd229f22e56c9 not found: ID does not exist" Nov 02 15:40:16.489392 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:16.489356 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b"] Nov 02 15:40:16.531477 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:16.531437 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-71d1eb-d7b559f8b-r994b"] Nov 02 15:40:17.166561 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:17.166522 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:17.465478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:17.465389 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:17.631960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:17.631926 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:18.106378 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:18.106329 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:18.457771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:18.457736 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0638183-955f-4411-8f7b-49c24124e8a6" path="/var/lib/kubelet/pods/d0638183-955f-4411-8f7b-49c24124e8a6/volumes" Nov 02 15:40:23.175101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:23.171182 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:25.042485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:25.042443 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:25.189784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:25.189747 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:25.382932 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:25.382896 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw"] Nov 02 15:40:25.383159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:25.383132 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" containerID="cri-o://abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188" gracePeriod=30 Nov 02 15:40:25.398364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:25.398335 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:27.699601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:27.699516 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:28.443907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:28.443870 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:28.717206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:28.717120 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:29.019651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.019571 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc"] Nov 02 15:40:29.020201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.020186 5433 cpu_manager.go:414] "containerMap: removing container" podUID="d0638183-955f-4411-8f7b-49c24124e8a6" containerName="maas-api" Nov 02 15:40:29.020251 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.020202 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0638183-955f-4411-8f7b-49c24124e8a6" containerName="maas-api" Nov 02 15:40:29.020331 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.020321 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0638183-955f-4411-8f7b-49c24124e8a6" containerName="maas-api" Nov 02 15:40:29.025861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.025838 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.084997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.076333 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fe8ef1-openshift-default-dockercfg-djfbx\"" Nov 02 15:40:29.117030 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.116995 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:29.179432 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.179390 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-certs\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.195473 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.179647 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-credential-socket\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.195919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.195819 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4120f89-4395-46f6-a62c-3fb79150b20e-istiod-ca-cert\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.196158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.196137 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-data\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.196394 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.196377 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnsm\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-kube-api-access-wfnsm\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.224008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.196501 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-podinfo\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.224008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.219115 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-token\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.224008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.219232 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-socket\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.224008 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.219265 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-envoy\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.243376 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.243293 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc"] Nov 02 15:40:29.321095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321005 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-credential-socket\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321063 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4120f89-4395-46f6-a62c-3fb79150b20e-istiod-ca-cert\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321108 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-data\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321185 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnsm\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-kube-api-access-wfnsm\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321204 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-podinfo\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321228 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-token\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321303 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-socket\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321332 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-envoy\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321365 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-certs\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321453 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-credential-socket\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321643 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-data\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321797 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-certs\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.321871 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.321849 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-socket\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.322273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.322246 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4120f89-4395-46f6-a62c-3fb79150b20e-istiod-ca-cert\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.323855 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.323828 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-envoy\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.324083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.324066 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-podinfo\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.381190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.381150 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-token\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.381339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.381150 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnsm\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-kube-api-access-wfnsm\") pod \"e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.422390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.422260 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:29.434221 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.434141 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:29.907483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.907458 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc"] Nov 02 15:40:29.910530 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:40:29.910496 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4120f89_4395_46f6_a62c_3fb79150b20e.slice/crio-71f9eb0229c2ca8925dba3ed1155d5c6a7c3f6c48d85a3b04d66af076b10311d WatchSource:0}: Error finding container 71f9eb0229c2ca8925dba3ed1155d5c6a7c3f6c48d85a3b04d66af076b10311d: Status 404 returned error can't find the container with id 71f9eb0229c2ca8925dba3ed1155d5c6a7c3f6c48d85a3b04d66af076b10311d Nov 02 15:40:29.913422 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.913387 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:29.913543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.913472 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:29.913543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:29.913521 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:30.401066 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.401029 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" event={"ID":"c4120f89-4395-46f6-a62c-3fb79150b20e","Type":"ContainerStarted","Data":"0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6"} Nov 02 15:40:30.401198 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.401102 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" event={"ID":"c4120f89-4395-46f6-a62c-3fb79150b20e","Type":"ContainerStarted","Data":"71f9eb0229c2ca8925dba3ed1155d5c6a7c3f6c48d85a3b04d66af076b10311d"} Nov 02 15:40:30.423392 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.423365 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:30.429442 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.429419 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:30.531552 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.531496 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:30.564867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.562101 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" podStartSLOduration=2.5620764129999998 podStartE2EDuration="2.562076413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:30.560748326 +0000 UTC m=+2534.233951522" watchObservedRunningTime="2026-11-02 15:40:30.562076413 +0000 UTC m=+2534.235279611" Nov 02 15:40:30.670683 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.670655 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:40:30.854129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854089 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-socket\") pod \"144b572d-c99c-4ea7-8b25-3a51bcd20421\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " Nov 02 15:40:30.854302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854181 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-podinfo\") pod \"144b572d-c99c-4ea7-8b25-3a51bcd20421\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " Nov 02 15:40:30.854302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854219 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-certs\") pod \"144b572d-c99c-4ea7-8b25-3a51bcd20421\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " Nov 02 15:40:30.854386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854329 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-envoy\") pod \"144b572d-c99c-4ea7-8b25-3a51bcd20421\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " Nov 02 15:40:30.854453 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854420 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-socket" pod "144b572d-c99c-4ea7-8b25-3a51bcd20421" (UID: "144b572d-c99c-4ea7-8b25-3a51bcd20421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.854525 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854430 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-credential-socket\") pod \"144b572d-c99c-4ea7-8b25-3a51bcd20421\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " Nov 02 15:40:30.854584 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854533 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-token\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-token\") pod \"144b572d-c99c-4ea7-8b25-3a51bcd20421\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " Nov 02 15:40:30.854584 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854574 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-kube-api-access-599mf\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-kube-api-access-599mf\") pod \"144b572d-c99c-4ea7-8b25-3a51bcd20421\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " Nov 02 15:40:30.854697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854622 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-data\") pod \"144b572d-c99c-4ea7-8b25-3a51bcd20421\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " Nov 02 15:40:30.854697 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854686 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144b572d-c99c-4ea7-8b25-3a51bcd20421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144b572d-c99c-4ea7-8b25-3a51bcd20421-istiod-ca-cert\") pod \"144b572d-c99c-4ea7-8b25-3a51bcd20421\" (UID: \"144b572d-c99c-4ea7-8b25-3a51bcd20421\") " Nov 02 15:40:30.854809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.854722 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-credential-socket" pod "144b572d-c99c-4ea7-8b25-3a51bcd20421" (UID: "144b572d-c99c-4ea7-8b25-3a51bcd20421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.855082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.855056 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-certs" pod "144b572d-c99c-4ea7-8b25-3a51bcd20421" (UID: "144b572d-c99c-4ea7-8b25-3a51bcd20421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.855255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.855210 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-data" pod "144b572d-c99c-4ea7-8b25-3a51bcd20421" (UID: "144b572d-c99c-4ea7-8b25-3a51bcd20421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.855476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.855390 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144b572d-c99c-4ea7-8b25-3a51bcd20421-istiod-ca-cert" pod "144b572d-c99c-4ea7-8b25-3a51bcd20421" (UID: "144b572d-c99c-4ea7-8b25-3a51bcd20421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:40:30.855567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.855511 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.855567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.855536 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.855567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.855551 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.857917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.857877 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-podinfo" pod "144b572d-c99c-4ea7-8b25-3a51bcd20421" (UID: "144b572d-c99c-4ea7-8b25-3a51bcd20421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:40:30.857917 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.857907 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-kube-api-access-599mf" pod "144b572d-c99c-4ea7-8b25-3a51bcd20421" (UID: "144b572d-c99c-4ea7-8b25-3a51bcd20421"). InnerVolumeSpecName "kube-api-access-599mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:30.858139 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.857939 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-token" pod "144b572d-c99c-4ea7-8b25-3a51bcd20421" (UID: "144b572d-c99c-4ea7-8b25-3a51bcd20421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:30.858356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.858333 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-envoy" pod "144b572d-c99c-4ea7-8b25-3a51bcd20421" (UID: "144b572d-c99c-4ea7-8b25-3a51bcd20421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.956596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.956513 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.956596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.956540 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.956596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.956552 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599mf\" (UniqueName: \"kubernetes.io/projected/144b572d-c99c-4ea7-8b25-3a51bcd20421-kube-api-access-599mf\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.956596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.956561 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.956596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.956569 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144b572d-c99c-4ea7-8b25-3a51bcd20421-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.956596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:30.956578 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144b572d-c99c-4ea7-8b25-3a51bcd20421-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.408924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.408892 5433 generic.go:356] "Generic (PLEG): container finished" podID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerID="abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188" exitCode=0 Nov 02 15:40:31.409100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.408993 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" Nov 02 15:40:31.409160 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.408986 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" event={"ID":"144b572d-c99c-4ea7-8b25-3a51bcd20421","Type":"ContainerDied","Data":"abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188"} Nov 02 15:40:31.409209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.409191 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw" event={"ID":"144b572d-c99c-4ea7-8b25-3a51bcd20421","Type":"ContainerDied","Data":"fe87e317852bef08538ebf8f4bbe91d11d2ba8d45a2b3a6558b8e50e214e10ae"} Nov 02 15:40:31.409254 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.409217 5433 scope.go:122] "RemoveContainer" containerID="abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188" Nov 02 15:40:31.409596 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.409573 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:31.411078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.411061 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:40:31.419271 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.419250 5433 scope.go:122] "RemoveContainer" containerID="8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec" Nov 02 15:40:31.428172 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.428154 5433 scope.go:122] "RemoveContainer" containerID="abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188" Nov 02 15:40:31.428465 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:40:31.428446 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188\": container with ID starting with abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188 not found: ID does not exist" containerID="abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188" Nov 02 15:40:31.428532 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.428473 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188"} err="failed to get container status \"abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188\": rpc error: code = NotFound desc = could not find container \"abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188\": container with ID starting with abe34374ec80e281cdb7cf4ef308f1e40a5881158ccae1628e9b1cc8af939188 not found: ID does not exist" Nov 02 15:40:31.428532 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.428489 5433 scope.go:122] "RemoveContainer" containerID="8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec" Nov 02 15:40:31.428753 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:40:31.428734 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec\": container with ID starting with 8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec not found: ID does not exist" containerID="8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec" Nov 02 15:40:31.428811 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.428763 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec"} err="failed to get container status \"8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec\": rpc error: code = NotFound desc = could not find container \"8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec\": container with ID starting with 8778e1d92eae8109fd3e67b897573706f08a26f2e3f69bd27a03d330313ad4ec not found: ID does not exist" Nov 02 15:40:31.678325 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.678252 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw"] Nov 02 15:40:31.685296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:31.685201 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-2n7dw"] Nov 02 15:40:32.186185 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:32.186136 5433 patch_prober.go:29] interesting pod/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:40:32.186672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:32.186236 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:40:32.457395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:32.457357 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" path="/var/lib/kubelet/pods/144b572d-c99c-4ea7-8b25-3a51bcd20421/volumes" Nov 02 15:40:33.425610 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:33.425576 5433 generic.go:356] "Generic (PLEG): container finished" podID="c978cb20-4d48-48db-a682-f8d9916a460e" containerID="b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249" exitCode=0 Nov 02 15:40:33.426021 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:33.425649 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" event={"ID":"c978cb20-4d48-48db-a682-f8d9916a460e","Type":"ContainerDied","Data":"b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249"} Nov 02 15:40:33.426224 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:33.426194 5433 scope.go:122] "RemoveContainer" containerID="b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249" Nov 02 15:40:33.427597 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:33.427572 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:33.427662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:33.427650 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:33.427698 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:33.427680 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:33.957459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:33.957418 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:34.185203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:34.185168 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:34.344067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:34.343956 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:34.448877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:34.448839 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" event={"ID":"c978cb20-4d48-48db-a682-f8d9916a460e","Type":"ContainerStarted","Data":"20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63"} Nov 02 15:40:34.957567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:34.957494 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:40:34.957771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:34.957728 5433 patch_prober.go:29] interesting pod/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:40:34.957831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:34.957789 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 15:40:35.957472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:35.957434 5433 patch_prober.go:29] interesting pod/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:40:35.958078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:35.957506 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 15:40:36.957743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:36.957708 5433 patch_prober.go:29] interesting pod/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:40:36.958182 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:36.957766 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 15:40:37.241891 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.241806 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:37.455372 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.455273 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:37.793565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.793528 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd"] Nov 02 15:40:37.847947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.847910 5433 cpu_manager.go:414] "containerMap: removing container" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" Nov 02 15:40:37.851721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.851680 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" Nov 02 15:40:37.852199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.852177 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" Nov 02 15:40:37.866259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.866232 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:37.884444 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.882385 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd"] Nov 02 15:40:37.943043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.942300 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-fe8ef1\"" Nov 02 15:40:37.958341 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.958310 5433 patch_prober.go:29] interesting pod/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:40:37.958805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.958361 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 15:40:37.975067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.975036 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eccaef4-511c-4792-8a2e-78fd5868d78a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd\" (UID: \"8eccaef4-511c-4792-8a2e-78fd5868d78a\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:37.975211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.975102 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gxc\" (UniqueName: \"kubernetes.io/projected/8eccaef4-511c-4792-8a2e-78fd5868d78a-kube-api-access-f6gxc\") pod \"payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd\" (UID: \"8eccaef4-511c-4792-8a2e-78fd5868d78a\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:37.988518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:37.988493 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-fe8ef1-dockercfg-2qsfm\"" Nov 02 15:40:38.076559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:38.076469 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eccaef4-511c-4792-8a2e-78fd5868d78a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd\" (UID: \"8eccaef4-511c-4792-8a2e-78fd5868d78a\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:38.076559 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:38.076530 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gxc\" (UniqueName: \"kubernetes.io/projected/8eccaef4-511c-4792-8a2e-78fd5868d78a-kube-api-access-f6gxc\") pod \"payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd\" (UID: \"8eccaef4-511c-4792-8a2e-78fd5868d78a\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:38.077250 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:38.077231 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eccaef4-511c-4792-8a2e-78fd5868d78a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd\" (UID: \"8eccaef4-511c-4792-8a2e-78fd5868d78a\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:38.142087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:38.142052 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gxc\" (UniqueName: \"kubernetes.io/projected/8eccaef4-511c-4792-8a2e-78fd5868d78a-kube-api-access-f6gxc\") pod \"payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd\" (UID: \"8eccaef4-511c-4792-8a2e-78fd5868d78a\") " pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:38.203638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:38.203601 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:38.606242 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:38.606196 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd"] Nov 02 15:40:38.608512 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:40:38.608480 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eccaef4_511c_4792_8a2e_78fd5868d78a.slice/crio-da1537e843149dc83adb1abd096126be0ce355ca57b3c7a8fafda3791c3051ee WatchSource:0}: Error finding container da1537e843149dc83adb1abd096126be0ce355ca57b3c7a8fafda3791c3051ee: Status 404 returned error can't find the container with id da1537e843149dc83adb1abd096126be0ce355ca57b3c7a8fafda3791c3051ee Nov 02 15:40:38.958065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:38.958031 5433 patch_prober.go:29] interesting pod/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:40:38.958268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:38.958090 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 15:40:39.484597 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:39.484553 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" event={"ID":"8eccaef4-511c-4792-8a2e-78fd5868d78a","Type":"ContainerStarted","Data":"1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377"} Nov 02 15:40:39.485135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:39.484622 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" event={"ID":"8eccaef4-511c-4792-8a2e-78fd5868d78a","Type":"ContainerStarted","Data":"da1537e843149dc83adb1abd096126be0ce355ca57b3c7a8fafda3791c3051ee"} Nov 02 15:40:39.485135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:39.484757 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:39.595533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:39.595470 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" podStartSLOduration=2.595453639 podStartE2EDuration="2.595453639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:39.593262172 +0000 UTC m=+2543.266465368" watchObservedRunningTime="2026-11-02 15:40:39.595453639 +0000 UTC m=+2543.268656826" Nov 02 15:40:39.962522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:39.962490 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:40:39.962775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:39.962756 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:40:39.963546 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:39.963530 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:40:40.184780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:40.184741 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:41.012165 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:40:41.012134 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:40:45.453874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:45.453840 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:46.742685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:46.742651 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:47.499456 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:47.499422 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:48.715553 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:48.715515 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:48.912861 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:48.912810 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:50.494088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:50.494060 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:40:51.005952 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:51.005897 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:51.177862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:51.177827 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:53.069660 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:53.069622 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:53.324905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:53.324808 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:53.557200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:53.557161 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:55.975997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:55.975007 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78"] Nov 02 15:40:56.042668 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.042638 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x"] Nov 02 15:40:56.154135 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.154105 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:56.361129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.361044 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:56.560023 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.548063 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:56.812278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.812239 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:40:56.938423 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.938379 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp"] Nov 02 15:40:56.939658 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.939626 5433 cpu_manager.go:414] "containerMap: removing container" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" Nov 02 15:40:56.939832 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.939819 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" Nov 02 15:40:56.944057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.940147 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="144b572d-c99c-4ea7-8b25-3a51bcd20421" containerName="istio-proxy" Nov 02 15:40:56.950992 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:56.948793 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:40:57.010999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.000362 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b1ee37-dockercfg-sj8st\"" Nov 02 15:40:57.010999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.000462 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b1ee37\"" Nov 02 15:40:57.028804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.013275 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp"] Nov 02 15:40:57.100592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.100502 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fad0c18-a050-45bf-be6f-a030762dc6b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp\" (UID: \"2fad0c18-a050-45bf-be6f-a030762dc6b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:40:57.100849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.100822 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psk44\" (UniqueName: \"kubernetes.io/projected/2fad0c18-a050-45bf-be6f-a030762dc6b5-kube-api-access-psk44\") pod \"payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp\" (UID: \"2fad0c18-a050-45bf-be6f-a030762dc6b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:40:57.201733 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.201688 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fad0c18-a050-45bf-be6f-a030762dc6b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp\" (UID: \"2fad0c18-a050-45bf-be6f-a030762dc6b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:40:57.201918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.201842 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psk44\" (UniqueName: \"kubernetes.io/projected/2fad0c18-a050-45bf-be6f-a030762dc6b5-kube-api-access-psk44\") pod \"payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp\" (UID: \"2fad0c18-a050-45bf-be6f-a030762dc6b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:40:57.202456 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.202436 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fad0c18-a050-45bf-be6f-a030762dc6b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp\" (UID: \"2fad0c18-a050-45bf-be6f-a030762dc6b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:40:57.259218 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.259187 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psk44\" (UniqueName: \"kubernetes.io/projected/2fad0c18-a050-45bf-be6f-a030762dc6b5-kube-api-access-psk44\") pod \"payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp\" (UID: \"2fad0c18-a050-45bf-be6f-a030762dc6b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:40:57.390149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.390115 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:40:57.765814 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:57.765785 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp"] Nov 02 15:40:57.767744 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:40:57.767711 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fad0c18_a050_45bf_be6f_a030762dc6b5.slice/crio-eef698f63a1ae76b7789ba3143145185056e1404282f8879e2944bf1052bd41b WatchSource:0}: Error finding container eef698f63a1ae76b7789ba3143145185056e1404282f8879e2944bf1052bd41b: Status 404 returned error can't find the container with id eef698f63a1ae76b7789ba3143145185056e1404282f8879e2944bf1052bd41b Nov 02 15:40:58.611206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:58.611166 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" event={"ID":"2fad0c18-a050-45bf-be6f-a030762dc6b5","Type":"ContainerStarted","Data":"9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac"} Nov 02 15:40:58.611664 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:58.611227 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" event={"ID":"2fad0c18-a050-45bf-be6f-a030762dc6b5","Type":"ContainerStarted","Data":"eef698f63a1ae76b7789ba3143145185056e1404282f8879e2944bf1052bd41b"} Nov 02 15:40:58.611664 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:58.611254 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:40:58.720940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:40:58.720872 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" podStartSLOduration=2.7208538410000003 podStartE2EDuration="2.720853841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:58.719822436 +0000 UTC m=+2562.393025631" watchObservedRunningTime="2026-11-02 15:40:58.720853841 +0000 UTC m=+2562.394057037" Nov 02 15:41:01.598700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:01.598666 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:02.395802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:02.395763 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:02.571803 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:02.571767 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:05.608587 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:05.607953 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:07.111729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.111686 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6"] Nov 02 15:41:07.112300 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.111941 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" podUID="93c9a70c-cb34-41bc-a936-0a9950627165" containerName="maas-api" containerID="cri-o://f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50" gracePeriod=30 Nov 02 15:41:07.213639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.213604 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:07.433113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.433086 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:41:07.515523 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.513444 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c9a70c-cb34-41bc-a936-0a9950627165-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c9a70c-cb34-41bc-a936-0a9950627165-openshift-service-ca\") pod \"93c9a70c-cb34-41bc-a936-0a9950627165\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " Nov 02 15:41:07.515523 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.513528 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls\") pod \"93c9a70c-cb34-41bc-a936-0a9950627165\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " Nov 02 15:41:07.515523 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.513611 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c9a70c-cb34-41bc-a936-0a9950627165-kube-api-access-wj8mp\" (UniqueName: \"kubernetes.io/projected/93c9a70c-cb34-41bc-a936-0a9950627165-kube-api-access-wj8mp\") pod \"93c9a70c-cb34-41bc-a936-0a9950627165\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " Nov 02 15:41:07.515523 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.513749 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-metrics-certs\") pod \"93c9a70c-cb34-41bc-a936-0a9950627165\" (UID: \"93c9a70c-cb34-41bc-a936-0a9950627165\") " Nov 02 15:41:07.532052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.529680 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c9a70c-cb34-41bc-a936-0a9950627165-openshift-service-ca" pod "93c9a70c-cb34-41bc-a936-0a9950627165" (UID: "93c9a70c-cb34-41bc-a936-0a9950627165"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:41:07.547003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.546753 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-metrics-certs" pod "93c9a70c-cb34-41bc-a936-0a9950627165" (UID: "93c9a70c-cb34-41bc-a936-0a9950627165"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:41:07.561065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.558168 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c9a70c-cb34-41bc-a936-0a9950627165-kube-api-access-wj8mp" pod "93c9a70c-cb34-41bc-a936-0a9950627165" (UID: "93c9a70c-cb34-41bc-a936-0a9950627165"). InnerVolumeSpecName "kube-api-access-wj8mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:07.561065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.558264 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls" pod "93c9a70c-cb34-41bc-a936-0a9950627165" (UID: "93c9a70c-cb34-41bc-a936-0a9950627165"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:41:07.615796 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.615765 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj8mp\" (UniqueName: \"kubernetes.io/projected/93c9a70c-cb34-41bc-a936-0a9950627165-kube-api-access-wj8mp\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:07.615796 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.615794 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:07.615796 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.615804 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c9a70c-cb34-41bc-a936-0a9950627165-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:07.616057 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.615813 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c9a70c-cb34-41bc-a936-0a9950627165-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:07.792018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.791908 5433 generic.go:356] "Generic (PLEG): container finished" podID="93c9a70c-cb34-41bc-a936-0a9950627165" containerID="f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50" exitCode=0 Nov 02 15:41:07.792018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.791967 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" Nov 02 15:41:07.792018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.792001 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" event={"ID":"93c9a70c-cb34-41bc-a936-0a9950627165","Type":"ContainerDied","Data":"f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50"} Nov 02 15:41:07.792294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.792068 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6" event={"ID":"93c9a70c-cb34-41bc-a936-0a9950627165","Type":"ContainerDied","Data":"d593a0f054f979deba7ebae36105affb3a8da0c562022bca7726ed905176fb01"} Nov 02 15:41:07.792294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.792096 5433 scope.go:122] "RemoveContainer" containerID="f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50" Nov 02 15:41:07.802505 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.802486 5433 scope.go:122] "RemoveContainer" containerID="f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50" Nov 02 15:41:07.802759 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:41:07.802739 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50\": container with ID starting with f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50 not found: ID does not exist" containerID="f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50" Nov 02 15:41:07.802805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.802767 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50"} err="failed to get container status \"f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50\": rpc error: code = NotFound desc = could not find container \"f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50\": container with ID starting with f0ea4af1d4d44568e62eda6d4d7c492279685730d0a70932ffb28d8eceb2da50 not found: ID does not exist" Nov 02 15:41:07.947097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.947067 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6"] Nov 02 15:41:07.949672 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:07.949650 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-61a835-75cb5476b5-pxxk6"] Nov 02 15:41:08.044998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:08.044915 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:08.192936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:08.192895 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:08.457184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:08.457149 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c9a70c-cb34-41bc-a936-0a9950627165" path="/var/lib/kubelet/pods/93c9a70c-cb34-41bc-a936-0a9950627165/volumes" Nov 02 15:41:08.866032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:08.865912 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:09.181527 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:09.181483 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:09.381347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:09.381313 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:09.620591 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:09.620516 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:41:09.785167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:09.785114 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:11.012244 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:41:11.012212 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:41:11.026795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.026763 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" containerID="cri-o://057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9" gracePeriod=45 Nov 02 15:41:11.072735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.072697 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" containerID="cri-o://0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2" gracePeriod=45 Nov 02 15:41:11.373025 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.373001 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:41:11.386091 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.386060 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:41:11.506122 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.506080 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-home\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-home\") pod \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " Nov 02 15:41:11.506634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.506609 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867c049b-fd84-4fe1-9e9a-939b8e387b8f-kube-api-access-skfd4\" (UniqueName: \"kubernetes.io/projected/867c049b-fd84-4fe1-9e9a-939b8e387b8f-kube-api-access-skfd4\") pod \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " Nov 02 15:41:11.506771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.506654 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-home\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-home\") pod \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " Nov 02 15:41:11.506771 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.506747 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tmp-dir\") pod \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " Nov 02 15:41:11.506906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.506803 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-kube-api-access-pkg5x\") pod \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " Nov 02 15:41:11.506906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.506552 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-home" pod "867c049b-fd84-4fe1-9e9a-939b8e387b8f" (UID: "867c049b-fd84-4fe1-9e9a-939b8e387b8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.507039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.506914 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-home" pod "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" (UID: "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.507318 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.507067 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tmp-dir" pod "867c049b-fd84-4fe1-9e9a-939b8e387b8f" (UID: "867c049b-fd84-4fe1-9e9a-939b8e387b8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.507505 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.507483 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tls-certs\") pod \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " Nov 02 15:41:11.507857 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.507836 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-model-cache\") pod \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " Nov 02 15:41:11.507949 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.507888 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tmp-dir\") pod \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " Nov 02 15:41:11.508024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.507960 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-dshm\") pod \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " Nov 02 15:41:11.508024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.508011 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tls-certs\") pod \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " Nov 02 15:41:11.508145 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.508040 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-model-cache\") pod \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\" (UID: \"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d\") " Nov 02 15:41:11.508145 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.508069 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-dshm\") pod \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\" (UID: \"867c049b-fd84-4fe1-9e9a-939b8e387b8f\") " Nov 02 15:41:11.508483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.508465 5433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-home\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.508483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.508485 5433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-home\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.508657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.508522 5433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tmp-dir\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.509444 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.509407 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-model-cache" pod "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" (UID: "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.509713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.509654 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-kube-api-access-pkg5x" pod "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" (UID: "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d"). InnerVolumeSpecName "kube-api-access-pkg5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:11.509713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.509692 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867c049b-fd84-4fe1-9e9a-939b8e387b8f-kube-api-access-skfd4" pod "867c049b-fd84-4fe1-9e9a-939b8e387b8f" (UID: "867c049b-fd84-4fe1-9e9a-939b8e387b8f"). InnerVolumeSpecName "kube-api-access-skfd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:11.510417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.510392 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tmp-dir" pod "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" (UID: "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.510616 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.510596 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-model-cache" pod "867c049b-fd84-4fe1-9e9a-939b8e387b8f" (UID: "867c049b-fd84-4fe1-9e9a-939b8e387b8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.510902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.510879 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-dshm" pod "867c049b-fd84-4fe1-9e9a-939b8e387b8f" (UID: "867c049b-fd84-4fe1-9e9a-939b8e387b8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.511713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.511688 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tls-certs" pod "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" (UID: "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:41:11.512119 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.512102 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-dshm" pod "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" (UID: "8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.512293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.512277 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tls-certs" pod "867c049b-fd84-4fe1-9e9a-939b8e387b8f" (UID: "867c049b-fd84-4fe1-9e9a-939b8e387b8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:41:11.610049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.609949 5433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tmp-dir\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.610049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.610003 5433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-dshm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.610049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.610020 5433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/867c049b-fd84-4fe1-9e9a-939b8e387b8f-tls-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.610049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.610036 5433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-model-cache\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.610049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.610049 5433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-dshm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.610385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.610064 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skfd4\" (UniqueName: \"kubernetes.io/projected/867c049b-fd84-4fe1-9e9a-939b8e387b8f-kube-api-access-skfd4\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.610385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.610080 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-kube-api-access-pkg5x\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.610385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.610095 5433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d-tls-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.610385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.610126 5433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867c049b-fd84-4fe1-9e9a-939b8e387b8f-model-cache\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.820623 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.820585 5433 generic.go:356] "Generic (PLEG): container finished" podID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerID="057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9" exitCode=0 Nov 02 15:41:11.820820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.820653 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" Nov 02 15:41:11.820820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.820673 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" event={"ID":"867c049b-fd84-4fe1-9e9a-939b8e387b8f","Type":"ContainerDied","Data":"057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9"} Nov 02 15:41:11.820820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.820716 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78" event={"ID":"867c049b-fd84-4fe1-9e9a-939b8e387b8f","Type":"ContainerDied","Data":"60f379b8ed59b34ead41995a45cb57a89457ab90ec83bc1e2dd808b01c033607"} Nov 02 15:41:11.820820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.820733 5433 scope.go:122] "RemoveContainer" containerID="057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9" Nov 02 15:41:11.822106 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.822078 5433 generic.go:356] "Generic (PLEG): container finished" podID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerID="0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2" exitCode=0 Nov 02 15:41:11.822255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.822106 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" event={"ID":"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d","Type":"ContainerDied","Data":"0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2"} Nov 02 15:41:11.822255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.822132 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" Nov 02 15:41:11.822255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.822140 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x" event={"ID":"8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d","Type":"ContainerDied","Data":"4c8c9fca1b9f7d4be814863a614ab485dca7e1e5d4297e27960542f4ad656196"} Nov 02 15:41:11.832455 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.832440 5433 scope.go:122] "RemoveContainer" containerID="057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9" Nov 02 15:41:11.832741 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:41:11.832718 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9\": container with ID starting with 057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9 not found: ID does not exist" containerID="057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9" Nov 02 15:41:11.832813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.832748 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9"} err="failed to get container status \"057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9\": rpc error: code = NotFound desc = could not find container \"057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9\": container with ID starting with 057a9c2fd51e79905b5dd8a42b36248ec2aee742d184ad58da8c6a395e48dbb9 not found: ID does not exist" Nov 02 15:41:11.832813 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.832771 5433 scope.go:122] "RemoveContainer" containerID="0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2" Nov 02 15:41:11.842837 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.842816 5433 scope.go:122] "RemoveContainer" containerID="0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2" Nov 02 15:41:11.843138 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:41:11.843118 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2\": container with ID starting with 0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2 not found: ID does not exist" containerID="0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2" Nov 02 15:41:11.843234 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.843143 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2"} err="failed to get container status \"0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2\": rpc error: code = NotFound desc = could not find container \"0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2\": container with ID starting with 0bd7e951b9739f8f2a33b370163823796592735b7cf2274b4ccf44d9843a60e2 not found: ID does not exist" Nov 02 15:41:11.978641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.978607 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78"] Nov 02 15:41:11.984310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:11.984280 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-w5-891723-kserve-675d8c7744-dlk78"] Nov 02 15:41:12.086076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:12.086047 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x"] Nov 02 15:41:12.090104 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:12.090080 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-891723/premium-w5-891723-kserve-7fc4dcb469-jnr2x"] Nov 02 15:41:12.458296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:12.458264 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" path="/var/lib/kubelet/pods/867c049b-fd84-4fe1-9e9a-939b8e387b8f/volumes" Nov 02 15:41:12.458764 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:12.458749 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" path="/var/lib/kubelet/pods/8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d/volumes" Nov 02 15:41:14.194339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.194299 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw"] Nov 02 15:41:14.195074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.195048 5433 cpu_manager.go:414] "containerMap: removing container" podUID="93c9a70c-cb34-41bc-a936-0a9950627165" containerName="maas-api" Nov 02 15:41:14.195074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.195069 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c9a70c-cb34-41bc-a936-0a9950627165" containerName="maas-api" Nov 02 15:41:14.195259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.195084 5433 cpu_manager.go:414] "containerMap: removing container" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" Nov 02 15:41:14.195259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.195091 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" Nov 02 15:41:14.195259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.195140 5433 cpu_manager.go:414] "containerMap: removing container" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" Nov 02 15:41:14.195259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.195145 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" Nov 02 15:41:14.195259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.195245 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="867c049b-fd84-4fe1-9e9a-939b8e387b8f" containerName="main" Nov 02 15:41:14.195259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.195260 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c9a70c-cb34-41bc-a936-0a9950627165" containerName="maas-api" Nov 02 15:41:14.195448 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.195267 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5a7ca4-ac8d-4597-bf7d-9693df3bba1d" containerName="main" Nov 02 15:41:14.200097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.200073 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.251386 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.251349 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e3b351-serving-cert\"" Nov 02 15:41:14.300315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.300284 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw"] Nov 02 15:41:14.341334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.341293 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.341334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.341336 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-metrics-certs\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.341588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.341374 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9nwk\" (UniqueName: \"kubernetes.io/projected/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-kube-api-access-z9nwk\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.341588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.341396 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.407884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.407850 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:14.476572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.476475 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.479804 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.479772 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-metrics-certs\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.480063 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.480042 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9nwk\" (UniqueName: \"kubernetes.io/projected/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-kube-api-access-z9nwk\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.480255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.480239 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.480485 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:41:14.480461 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-serving-cert: secret "maas-api-e2e-api-b-e3b351-serving-cert" not found Nov 02 15:41:14.480576 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:41:14.480562 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls podName:c4a51c2d-c28e-4145-91ca-2e3f04f8101c nodeName:}" failed. No retries permitted until 2026-11-02 15:41:14.980539836 +0000 UTC m=+2578.653743031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls") pod "maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" (UID: "c4a51c2d-c28e-4145-91ca-2e3f04f8101c") : secret "maas-api-e2e-api-b-e3b351-serving-cert" not found Nov 02 15:41:14.500535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.500500 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.508361 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.506508 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-metrics-certs\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.538342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.538316 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9nwk\" (UniqueName: \"kubernetes.io/projected/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-kube-api-access-z9nwk\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.575516 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.575486 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:14.721712 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.721681 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:14.987668 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.987634 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:14.990242 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:14.990217 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls\") pod \"maas-api-e2e-api-b-e3b351-8496ff996c-v69sw\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:15.114094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:15.114064 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:15.484276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:15.484249 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw"] Nov 02 15:41:15.486042 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:41:15.486004 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a51c2d_c28e_4145_91ca_2e3f04f8101c.slice/crio-a53b7108a8e83081d359f897c927000a75e74bf22020888d07b79a510ed5b6ce WatchSource:0}: Error finding container a53b7108a8e83081d359f897c927000a75e74bf22020888d07b79a510ed5b6ce: Status 404 returned error can't find the container with id a53b7108a8e83081d359f897c927000a75e74bf22020888d07b79a510ed5b6ce Nov 02 15:41:15.487705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:15.487689 5433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:41:15.734572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:15.734480 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:15.860170 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:15.860128 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" event={"ID":"c4a51c2d-c28e-4145-91ca-2e3f04f8101c","Type":"ContainerStarted","Data":"a53b7108a8e83081d359f897c927000a75e74bf22020888d07b79a510ed5b6ce"} Nov 02 15:41:15.876900 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:15.876868 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:16.142653 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:16.142621 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:16.868376 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:16.868285 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" event={"ID":"c4a51c2d-c28e-4145-91ca-2e3f04f8101c","Type":"ContainerStarted","Data":"b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836"} Nov 02 15:41:16.868376 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:16.868353 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:16.980130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:16.980083 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" podStartSLOduration=2.031478086 podStartE2EDuration="2.980069242s" totalImagesPullingTime="948.591156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:16.977710235 +0000 UTC m=+2580.650913429" watchObservedRunningTime="2026-11-02 15:41:16.980069242 +0000 UTC m=+2580.653272436" Nov 02 15:41:19.851843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:19.851805 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:22.881320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:22.881290 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:41:24.995065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:24.995031 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:30.055272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:30.055238 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:30.858091 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:30.858057 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:34.360702 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:34.360660 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:34.503679 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:34.503648 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:34.650113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:34.650080 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:35.141939 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:35.141902 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:40.366472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:40.366438 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:40.521408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:40.521378 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:41.011948 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:41:41.011912 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:41:45.459894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:45.459857 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:50.232455 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.232420 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv"] Nov 02 15:41:50.237257 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.237236 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.285911 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.285884 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fe8ef1-serving-cert\"" Nov 02 15:41:50.319225 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.319188 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kf7z\" (UniqueName: \"kubernetes.io/projected/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-kube-api-access-2kf7z\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.319412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.319250 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-openshift-service-ca\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.319464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.319416 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-metrics-certs\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.319506 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.319462 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.334716 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.334685 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv"] Nov 02 15:41:50.420709 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.420669 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-metrics-certs\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.420874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.420727 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.420874 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:41:50.420835 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-serving-cert: secret "maas-api-e2e-api-b-fe8ef1-serving-cert" not found Nov 02 15:41:50.420874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.420865 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kf7z\" (UniqueName: \"kubernetes.io/projected/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-kube-api-access-2kf7z\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.421061 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.420899 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-openshift-service-ca\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.421061 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:41:50.420921 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls podName:440458b0-3f93-4f8a-a9b8-8f0e321bc22c nodeName:}" failed. No retries permitted until 2026-11-02 15:41:50.920899427 +0000 UTC m=+2614.594102621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls") pod "maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" (UID: "440458b0-3f93-4f8a-a9b8-8f0e321bc22c") : secret "maas-api-e2e-api-b-fe8ef1-serving-cert" not found Nov 02 15:41:50.421515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.421492 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-openshift-service-ca\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.423178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.423155 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-metrics-certs\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.433849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.433823 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:50.525060 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.519877 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kf7z\" (UniqueName: \"kubernetes.io/projected/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-kube-api-access-2kf7z\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.609138 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.609102 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:50.754556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.754521 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:50.927261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.927218 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:50.929719 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:50.929695 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls\") pod \"maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:51.152478 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:51.152447 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:51.522838 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:51.522813 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv"] Nov 02 15:41:51.524783 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:41:51.524740 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-08846d32c6a8b9e81127830d8c3878748d3120076db5c86f037e5a12aee75db0 WatchSource:0}: Error finding container 08846d32c6a8b9e81127830d8c3878748d3120076db5c86f037e5a12aee75db0: Status 404 returned error can't find the container with id 08846d32c6a8b9e81127830d8c3878748d3120076db5c86f037e5a12aee75db0 Nov 02 15:41:52.117726 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:52.117693 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" event={"ID":"440458b0-3f93-4f8a-a9b8-8f0e321bc22c","Type":"ContainerStarted","Data":"08846d32c6a8b9e81127830d8c3878748d3120076db5c86f037e5a12aee75db0"} Nov 02 15:41:53.124846 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:53.124805 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" event={"ID":"440458b0-3f93-4f8a-a9b8-8f0e321bc22c","Type":"ContainerStarted","Data":"724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa"} Nov 02 15:41:53.125344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:53.124871 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:41:53.236812 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:53.236748 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" podStartSLOduration=2.220229054 podStartE2EDuration="3.236728963s" totalImagesPullingTime="1.016499909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:53.235164534 +0000 UTC m=+2616.908367733" watchObservedRunningTime="2026-11-02 15:41:53.236728963 +0000 UTC m=+2616.909932159" Nov 02 15:41:55.704040 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:55.704000 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:41:59.136608 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:41:59.136530 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:42:00.474896 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:00.474859 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:00.984914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:00.984882 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:06.166327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:06.166291 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:06.331901 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:06.331826 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:06.526334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:06.526252 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:11.012019 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:42:11.011960 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:42:11.397156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:11.397100 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:14.361898 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:14.361858 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:16.476432 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:16.476399 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:19.606098 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:19.606052 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:20.299180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:20.299107 5433 generic.go:356] "Generic (PLEG): container finished" podID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerID="0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6" exitCode=0 Nov 02 15:42:20.299363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:20.299182 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" event={"ID":"c4120f89-4395-46f6-a62c-3fb79150b20e","Type":"ContainerDied","Data":"0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6"} Nov 02 15:42:20.299584 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:20.299568 5433 scope.go:122] "RemoveContainer" containerID="0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6" Nov 02 15:42:20.301142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:20.301113 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:42:20.301268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:20.301195 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:42:20.301268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:20.301243 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:42:21.307720 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:21.307684 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" event={"ID":"c4120f89-4395-46f6-a62c-3fb79150b20e","Type":"ContainerStarted","Data":"80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e"} Nov 02 15:42:21.440001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:21.439421 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:42:21.440001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:21.439542 5433 patch_prober.go:29] interesting pod/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 15:42:21.440001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:21.439587 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 15:42:21.486294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:21.486263 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:22.441113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:22.441079 5433 patch_prober.go:29] interesting pod/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 15:42:22.441529 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:22.441131 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 15:42:23.429001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:23.425570 5433 patch_prober.go:29] interesting pod/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 15:42:23.429001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:23.425651 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 15:42:24.429413 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:24.429370 5433 patch_prober.go:29] interesting pod/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 15:42:24.430023 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:24.429988 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 15:42:25.433998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:25.429338 5433 patch_prober.go:29] interesting pod/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 15:42:25.433998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:25.429399 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 15:42:26.431293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:26.431256 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:42:26.432216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:26.431969 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:42:26.432686 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:26.432663 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:42:26.520568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:26.520523 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:31.191761 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:31.191713 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:31.574459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:31.574324 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:36.670631 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:36.670581 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:41.012432 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:42:41.012395 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:42:42.054594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:42.054554 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:42.373339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:42.373258 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:47.451645 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:47.451606 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:52.509969 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:52.509929 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:52.653450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:52.653417 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:52.909028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:52.908971 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:55.156002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:55.155941 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8"] Nov 02 15:42:55.527842 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:55.527760 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:55.676735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:55.676700 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:42:58.390130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:42:58.390098 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:00.157404 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.157346 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" podUID="b54066c7-e857-4a05-a72c-f194d6a6b655" containerName="payload-processing" containerID="cri-o://2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134" gracePeriod=25 Nov 02 15:43:00.428187 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.428155 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:43:00.518819 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.518790 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54066c7-e857-4a05-a72c-f194d6a6b655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54066c7-e857-4a05-a72c-f194d6a6b655-plugins-config-volume\") pod \"b54066c7-e857-4a05-a72c-f194d6a6b655\" (UID: \"b54066c7-e857-4a05-a72c-f194d6a6b655\") " Nov 02 15:43:00.519046 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.518876 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54066c7-e857-4a05-a72c-f194d6a6b655-kube-api-access-gr24z\" (UniqueName: \"kubernetes.io/projected/b54066c7-e857-4a05-a72c-f194d6a6b655-kube-api-access-gr24z\") pod \"b54066c7-e857-4a05-a72c-f194d6a6b655\" (UID: \"b54066c7-e857-4a05-a72c-f194d6a6b655\") " Nov 02 15:43:00.519195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.519169 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54066c7-e857-4a05-a72c-f194d6a6b655-plugins-config-volume" pod "b54066c7-e857-4a05-a72c-f194d6a6b655" (UID: "b54066c7-e857-4a05-a72c-f194d6a6b655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:00.519403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.519382 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54066c7-e857-4a05-a72c-f194d6a6b655-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:00.521120 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.521089 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54066c7-e857-4a05-a72c-f194d6a6b655-kube-api-access-gr24z" pod "b54066c7-e857-4a05-a72c-f194d6a6b655" (UID: "b54066c7-e857-4a05-a72c-f194d6a6b655"). InnerVolumeSpecName "kube-api-access-gr24z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:00.568203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.568174 5433 generic.go:356] "Generic (PLEG): container finished" podID="b54066c7-e857-4a05-a72c-f194d6a6b655" containerID="2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134" exitCode=0 Nov 02 15:43:00.568380 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.568244 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" Nov 02 15:43:00.568380 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.568265 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" event={"ID":"b54066c7-e857-4a05-a72c-f194d6a6b655","Type":"ContainerDied","Data":"2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134"} Nov 02 15:43:00.568380 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.568338 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8" event={"ID":"b54066c7-e857-4a05-a72c-f194d6a6b655","Type":"ContainerDied","Data":"ba8bee49556e6b3bff7cc3e877e1b925973d36de7aec2299a9019a1c7a569e5a"} Nov 02 15:43:00.568380 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.568371 5433 scope.go:122] "RemoveContainer" containerID="2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134" Nov 02 15:43:00.580468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.580446 5433 scope.go:122] "RemoveContainer" containerID="2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134" Nov 02 15:43:00.580776 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:43:00.580754 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134\": container with ID starting with 2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134 not found: ID does not exist" containerID="2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134" Nov 02 15:43:00.580851 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.580787 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134"} err="failed to get container status \"2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134\": rpc error: code = NotFound desc = could not find container \"2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134\": container with ID starting with 2ddd69abeebabdee4e8dad76af37d2fb7b472c9e829d479de090c6c2ffa5c134 not found: ID does not exist" Nov 02 15:43:00.620955 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.620923 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr24z\" (UniqueName: \"kubernetes.io/projected/b54066c7-e857-4a05-a72c-f194d6a6b655-kube-api-access-gr24z\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:00.723788 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.723695 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8"] Nov 02 15:43:00.727094 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:00.727063 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c35ee-79f4678fdf-w4zq8"] Nov 02 15:43:02.457044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:02.457010 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54066c7-e857-4a05-a72c-f194d6a6b655" path="/var/lib/kubelet/pods/b54066c7-e857-4a05-a72c-f194d6a6b655/volumes" Nov 02 15:43:03.468086 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:03.468051 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:08.598772 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:08.598735 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:11.011588 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:43:11.011554 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:43:13.901323 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:13.901291 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:17.229555 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:17.229520 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:19.748172 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:19.748134 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:20.074059 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:20.073939 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:25.395712 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:25.395678 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:30.121576 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.121331 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz"] Nov 02 15:43:30.136906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.136823 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" podUID="1193425a-4b6b-47ac-827f-20a00f789df1" containerName="maas-api" containerID="cri-o://406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1" gracePeriod=30 Nov 02 15:43:30.291577 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.291542 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:30.449287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.449254 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:43:30.536616 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.536587 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1193425a-4b6b-47ac-827f-20a00f789df1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1193425a-4b6b-47ac-827f-20a00f789df1-openshift-service-ca\") pod \"1193425a-4b6b-47ac-827f-20a00f789df1\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " Nov 02 15:43:30.536816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.536632 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1193425a-4b6b-47ac-827f-20a00f789df1-kube-api-access-gbcmc\" (UniqueName: \"kubernetes.io/projected/1193425a-4b6b-47ac-827f-20a00f789df1-kube-api-access-gbcmc\") pod \"1193425a-4b6b-47ac-827f-20a00f789df1\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " Nov 02 15:43:30.536816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.536765 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-metrics-certs\") pod \"1193425a-4b6b-47ac-827f-20a00f789df1\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " Nov 02 15:43:30.536816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.536800 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls\") pod \"1193425a-4b6b-47ac-827f-20a00f789df1\" (UID: \"1193425a-4b6b-47ac-827f-20a00f789df1\") " Nov 02 15:43:30.537018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.536967 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1193425a-4b6b-47ac-827f-20a00f789df1-openshift-service-ca" pod "1193425a-4b6b-47ac-827f-20a00f789df1" (UID: "1193425a-4b6b-47ac-827f-20a00f789df1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:30.537414 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.537381 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1193425a-4b6b-47ac-827f-20a00f789df1-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:30.538831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.538804 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls" pod "1193425a-4b6b-47ac-827f-20a00f789df1" (UID: "1193425a-4b6b-47ac-827f-20a00f789df1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:43:30.538924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.538866 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-metrics-certs" pod "1193425a-4b6b-47ac-827f-20a00f789df1" (UID: "1193425a-4b6b-47ac-827f-20a00f789df1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:43:30.538924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.538870 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1193425a-4b6b-47ac-827f-20a00f789df1-kube-api-access-gbcmc" pod "1193425a-4b6b-47ac-827f-20a00f789df1" (UID: "1193425a-4b6b-47ac-827f-20a00f789df1"). InnerVolumeSpecName "kube-api-access-gbcmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:30.638701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.638613 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbcmc\" (UniqueName: \"kubernetes.io/projected/1193425a-4b6b-47ac-827f-20a00f789df1-kube-api-access-gbcmc\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:30.638701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.638647 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:30.638701 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.638657 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1193425a-4b6b-47ac-827f-20a00f789df1-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:30.831001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.829571 5433 generic.go:356] "Generic (PLEG): container finished" podID="1193425a-4b6b-47ac-827f-20a00f789df1" containerID="406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1" exitCode=0 Nov 02 15:43:30.831001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.829668 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" event={"ID":"1193425a-4b6b-47ac-827f-20a00f789df1","Type":"ContainerDied","Data":"406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1"} Nov 02 15:43:30.831001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.829716 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" event={"ID":"1193425a-4b6b-47ac-827f-20a00f789df1","Type":"ContainerDied","Data":"7385844c846b8193ed0d1e6a54f71f7631060b43c79c2dcdd701bbed4c95f989"} Nov 02 15:43:30.831001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.829745 5433 scope.go:122] "RemoveContainer" containerID="406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1" Nov 02 15:43:30.831001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.829908 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz" Nov 02 15:43:30.858625 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.858597 5433 scope.go:122] "RemoveContainer" containerID="406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1" Nov 02 15:43:30.858999 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:43:30.858963 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1\": container with ID starting with 406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1 not found: ID does not exist" containerID="406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1" Nov 02 15:43:30.859074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.859011 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1"} err="failed to get container status \"406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1\": rpc error: code = NotFound desc = could not find container \"406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1\": container with ID starting with 406f5b9dd9349d9f864f80627eb7c256443dd4006b5bbc6a27fd61b9f1e581b1 not found: ID does not exist" Nov 02 15:43:30.988450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:30.988372 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz"] Nov 02 15:43:31.034546 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:31.034512 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-891723-7dcfb5cf48-b76zz"] Nov 02 15:43:31.519031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:31.518998 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:32.147765 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:32.147712 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:32.457101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:32.457027 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1193425a-4b6b-47ac-827f-20a00f789df1" path="/var/lib/kubelet/pods/1193425a-4b6b-47ac-827f-20a00f789df1/volumes" Nov 02 15:43:32.560398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:32.560367 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:32.784440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:32.784354 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:33.491622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:33.491562 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:33.662524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:33.662488 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:33.854174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:33.854088 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:36.649905 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:36.649872 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:38.266258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:38.266216 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:40.310307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:40.310276 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:40.480402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:40.480369 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:40.708789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:40.708752 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:41.012196 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:43:41.012109 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:43:43.912055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:43.912016 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:49.374109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:49.374069 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2"] Nov 02 15:43:49.529095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:49.529062 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4"] Nov 02 15:43:49.636165 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:49.630176 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:49.822997 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:49.817444 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:49.990724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:49.990630 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:54.375078 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.375030 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" podUID="b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca" containerName="payload-processing" containerID="cri-o://a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97" gracePeriod=25 Nov 02 15:43:54.529360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.529323 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" podUID="95cc3a4b-68fe-42f3-b931-70b6d1132b05" containerName="payload-pre-processing" containerID="cri-o://6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d" gracePeriod=25 Nov 02 15:43:54.664377 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.664350 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:43:54.766461 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.766421 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-plugins-config-volume\") pod \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\" (UID: \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\") " Nov 02 15:43:54.766651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.766531 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-kube-api-access-nt75c\" (UniqueName: \"kubernetes.io/projected/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-kube-api-access-nt75c\") pod \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\" (UID: \"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca\") " Nov 02 15:43:54.766864 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.766833 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-plugins-config-volume" pod "b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca" (UID: "b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:54.767235 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.767210 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:54.768695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.768669 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-kube-api-access-nt75c" pod "b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca" (UID: "b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca"). InnerVolumeSpecName "kube-api-access-nt75c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:54.789690 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.789664 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:43:54.868284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.868246 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95cc3a4b-68fe-42f3-b931-70b6d1132b05-kube-api-access-bkr8k\" (UniqueName: \"kubernetes.io/projected/95cc3a4b-68fe-42f3-b931-70b6d1132b05-kube-api-access-bkr8k\") pod \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\" (UID: \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\") " Nov 02 15:43:54.868482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.868358 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95cc3a4b-68fe-42f3-b931-70b6d1132b05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95cc3a4b-68fe-42f3-b931-70b6d1132b05-plugins-config-volume\") pod \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\" (UID: \"95cc3a4b-68fe-42f3-b931-70b6d1132b05\") " Nov 02 15:43:54.868833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.868812 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt75c\" (UniqueName: \"kubernetes.io/projected/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca-kube-api-access-nt75c\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:54.868938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.868886 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95cc3a4b-68fe-42f3-b931-70b6d1132b05-plugins-config-volume" pod "95cc3a4b-68fe-42f3-b931-70b6d1132b05" (UID: "95cc3a4b-68fe-42f3-b931-70b6d1132b05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:54.870506 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.870486 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cc3a4b-68fe-42f3-b931-70b6d1132b05-kube-api-access-bkr8k" pod "95cc3a4b-68fe-42f3-b931-70b6d1132b05" (UID: "95cc3a4b-68fe-42f3-b931-70b6d1132b05"). InnerVolumeSpecName "kube-api-access-bkr8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:54.975002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.970944 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkr8k\" (UniqueName: \"kubernetes.io/projected/95cc3a4b-68fe-42f3-b931-70b6d1132b05-kube-api-access-bkr8k\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:54.975002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:54.971004 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95cc3a4b-68fe-42f3-b931-70b6d1132b05-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:55.014316 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.014283 5433 generic.go:356] "Generic (PLEG): container finished" podID="b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca" containerID="a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97" exitCode=0 Nov 02 15:43:55.014508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.014332 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" Nov 02 15:43:55.014508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.014370 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" event={"ID":"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca","Type":"ContainerDied","Data":"a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97"} Nov 02 15:43:55.014508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.014421 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2" event={"ID":"b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca","Type":"ContainerDied","Data":"a6e641de7c6212a570ba3820d02fcaef36742431267ec824be88e9fd45e85171"} Nov 02 15:43:55.014508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.014450 5433 scope.go:122] "RemoveContainer" containerID="a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97" Nov 02 15:43:55.016467 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.016437 5433 generic.go:356] "Generic (PLEG): container finished" podID="95cc3a4b-68fe-42f3-b931-70b6d1132b05" containerID="6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d" exitCode=0 Nov 02 15:43:55.016569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.016520 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" event={"ID":"95cc3a4b-68fe-42f3-b931-70b6d1132b05","Type":"ContainerDied","Data":"6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d"} Nov 02 15:43:55.016604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.016566 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" event={"ID":"95cc3a4b-68fe-42f3-b931-70b6d1132b05","Type":"ContainerDied","Data":"5ac5a6f424c993c46d0a1a7bab3c2d27b2867e0ed9f47f7b2197acc49bdd0cc9"} Nov 02 15:43:55.016656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.016637 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4" Nov 02 15:43:55.028885 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.028760 5433 scope.go:122] "RemoveContainer" containerID="a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97" Nov 02 15:43:55.029311 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:43:55.029284 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97\": container with ID starting with a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97 not found: ID does not exist" containerID="a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97" Nov 02 15:43:55.029385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.029323 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97"} err="failed to get container status \"a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97\": rpc error: code = NotFound desc = could not find container \"a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97\": container with ID starting with a7d1b01419f296cb66d34745e96706b40a64f07f48b5ada21e3df5be06689c97 not found: ID does not exist" Nov 02 15:43:55.029385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.029346 5433 scope.go:122] "RemoveContainer" containerID="6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d" Nov 02 15:43:55.039695 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.039669 5433 scope.go:122] "RemoveContainer" containerID="6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d" Nov 02 15:43:55.040053 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:43:55.039965 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d\": container with ID starting with 6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d not found: ID does not exist" containerID="6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d" Nov 02 15:43:55.040053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.040010 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d"} err="failed to get container status \"6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d\": rpc error: code = NotFound desc = could not find container \"6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d\": container with ID starting with 6ab3f1ef3abe01f30e211bd5c952d796f5333544a3d4af453bfb1e667d3ae38d not found: ID does not exist" Nov 02 15:43:55.058393 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.058366 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:55.176175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.176137 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4"] Nov 02 15:43:55.180647 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.180622 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb6bd4-6c7cd56875-sdch4"] Nov 02 15:43:55.282217 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.282125 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2"] Nov 02 15:43:55.284621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:55.284593 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb6bd4-685b7bb6b9-rzwr2"] Nov 02 15:43:56.458041 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:56.458000 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95cc3a4b-68fe-42f3-b931-70b6d1132b05" path="/var/lib/kubelet/pods/95cc3a4b-68fe-42f3-b931-70b6d1132b05/volumes" Nov 02 15:43:56.458635 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:56.458612 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca" path="/var/lib/kubelet/pods/b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca/volumes" Nov 02 15:43:57.892290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:57.892252 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:43:58.077087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:43:58.077050 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:00.292089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:00.292051 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:05.488918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:05.488880 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:10.950000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:10.946712 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:11.012420 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:44:11.012386 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:44:16.181469 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:16.181435 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:21.088836 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:21.088797 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:21.241533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:21.241496 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:21.427735 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:21.427662 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:21.598606 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:21.598565 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz"] Nov 02 15:44:21.598843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:21.598816 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" containerID="cri-o://de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1" gracePeriod=30 Nov 02 15:44:26.528542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.528511 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:26.870593 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.870569 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:44:26.964766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.964739 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-envoy\") pod \"76eb87da-7c2e-4924-b606-3f97d446fc21\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " Nov 02 15:44:26.964946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.964784 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-podinfo\") pod \"76eb87da-7c2e-4924-b606-3f97d446fc21\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " Nov 02 15:44:26.964946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.964814 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76eb87da-7c2e-4924-b606-3f97d446fc21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb87da-7c2e-4924-b606-3f97d446fc21-istiod-ca-cert\") pod \"76eb87da-7c2e-4924-b606-3f97d446fc21\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " Nov 02 15:44:26.964946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.964849 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-data\") pod \"76eb87da-7c2e-4924-b606-3f97d446fc21\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " Nov 02 15:44:26.964946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.964885 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-token\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-token\") pod \"76eb87da-7c2e-4924-b606-3f97d446fc21\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " Nov 02 15:44:26.964946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.964945 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-socket\") pod \"76eb87da-7c2e-4924-b606-3f97d446fc21\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " Nov 02 15:44:26.965270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965033 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-certs\") pod \"76eb87da-7c2e-4924-b606-3f97d446fc21\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " Nov 02 15:44:26.965270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965065 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-kube-api-access-2zpdk\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-kube-api-access-2zpdk\") pod \"76eb87da-7c2e-4924-b606-3f97d446fc21\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " Nov 02 15:44:26.965270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965152 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-credential-socket\") pod \"76eb87da-7c2e-4924-b606-3f97d446fc21\" (UID: \"76eb87da-7c2e-4924-b606-3f97d446fc21\") " Nov 02 15:44:26.965270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965251 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-data" pod "76eb87da-7c2e-4924-b606-3f97d446fc21" (UID: "76eb87da-7c2e-4924-b606-3f97d446fc21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:26.965494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965317 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-socket" pod "76eb87da-7c2e-4924-b606-3f97d446fc21" (UID: "76eb87da-7c2e-4924-b606-3f97d446fc21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:26.965494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965399 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-certs" pod "76eb87da-7c2e-4924-b606-3f97d446fc21" (UID: "76eb87da-7c2e-4924-b606-3f97d446fc21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:26.965605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965503 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76eb87da-7c2e-4924-b606-3f97d446fc21-istiod-ca-cert" pod "76eb87da-7c2e-4924-b606-3f97d446fc21" (UID: "76eb87da-7c2e-4924-b606-3f97d446fc21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:44:26.965671 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965619 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-credential-socket" pod "76eb87da-7c2e-4924-b606-3f97d446fc21" (UID: "76eb87da-7c2e-4924-b606-3f97d446fc21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:26.965958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965935 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb87da-7c2e-4924-b606-3f97d446fc21-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.966087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965961 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.966087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.965990 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.966087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.966008 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.966087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.966023 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.967114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.967091 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-podinfo" pod "76eb87da-7c2e-4924-b606-3f97d446fc21" (UID: "76eb87da-7c2e-4924-b606-3f97d446fc21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:44:26.967285 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.967263 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-envoy" pod "76eb87da-7c2e-4924-b606-3f97d446fc21" (UID: "76eb87da-7c2e-4924-b606-3f97d446fc21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:26.967333 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.967317 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-token" pod "76eb87da-7c2e-4924-b606-3f97d446fc21" (UID: "76eb87da-7c2e-4924-b606-3f97d446fc21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:26.967537 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:26.967514 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-kube-api-access-2zpdk" pod "76eb87da-7c2e-4924-b606-3f97d446fc21" (UID: "76eb87da-7c2e-4924-b606-3f97d446fc21"). InnerVolumeSpecName "kube-api-access-2zpdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:27.067862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.067825 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:27.067862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.067857 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zpdk\" (UniqueName: \"kubernetes.io/projected/76eb87da-7c2e-4924-b606-3f97d446fc21-kube-api-access-2zpdk\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:27.067862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.067868 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:27.068116 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.067878 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb87da-7c2e-4924-b606-3f97d446fc21-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:27.216307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.216273 5433 generic.go:356] "Generic (PLEG): container finished" podID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerID="de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1" exitCode=0 Nov 02 15:44:27.216500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.216325 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" Nov 02 15:44:27.216500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.216340 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" event={"ID":"76eb87da-7c2e-4924-b606-3f97d446fc21","Type":"ContainerDied","Data":"de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1"} Nov 02 15:44:27.216500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.216382 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz" event={"ID":"76eb87da-7c2e-4924-b606-3f97d446fc21","Type":"ContainerDied","Data":"1d4422d2ba65b39f3eedabcab067edbd679093e2b254bb4324bb802868bf20c1"} Nov 02 15:44:27.216500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.216403 5433 scope.go:122] "RemoveContainer" containerID="de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1" Nov 02 15:44:27.227925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.227907 5433 scope.go:122] "RemoveContainer" containerID="e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9" Nov 02 15:44:27.236822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.236798 5433 scope.go:122] "RemoveContainer" containerID="de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1" Nov 02 15:44:27.237137 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:44:27.237116 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1\": container with ID starting with de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1 not found: ID does not exist" containerID="de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1" Nov 02 15:44:27.237191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.237148 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1"} err="failed to get container status \"de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1\": rpc error: code = NotFound desc = could not find container \"de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1\": container with ID starting with de21dcb09cc020b1c758e88c1de5f6387867ecc272f662399ebdfaf61e32e2c1 not found: ID does not exist" Nov 02 15:44:27.237191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.237167 5433 scope.go:122] "RemoveContainer" containerID="e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9" Nov 02 15:44:27.237429 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:44:27.237412 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9\": container with ID starting with e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9 not found: ID does not exist" containerID="e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9" Nov 02 15:44:27.237470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.237438 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9"} err="failed to get container status \"e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9\": rpc error: code = NotFound desc = could not find container \"e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9\": container with ID starting with e089cd9381601387c041307a54ccc2f6c8820ecd6bbf7e67c29c2af5b2e058f9 not found: ID does not exist" Nov 02 15:44:27.401126 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.401088 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz"] Nov 02 15:44:27.403484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:27.403455 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-hv4mz"] Nov 02 15:44:28.457182 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:28.457149 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" path="/var/lib/kubelet/pods/76eb87da-7c2e-4924-b606-3f97d446fc21/volumes" Nov 02 15:44:31.867125 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:31.867088 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:32.048837 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:32.048797 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:37.345742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:37.345670 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:41.012568 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:44:41.012528 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:44:41.927881 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:41.927837 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:43.798144 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:43.798110 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:44.090571 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:44.090492 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:44.273799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:44.273757 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:44.823384 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:44.823351 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:49.686822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:49.686782 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:51.627738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:51.627703 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:51.892213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:51.892127 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:52.136445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:52.136411 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7"] Nov 02 15:44:52.136659 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:52.136607 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" containerID="cri-o://67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da" gracePeriod=30 Nov 02 15:44:52.178449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:52.178371 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:53.954878 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:53.954835 5433 patch_prober.go:29] interesting pod/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:44:53.955291 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:53.954902 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:44:55.381505 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:55.381467 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:44:57.404472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.404403 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:44:57.407028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.407002 5433 generic.go:356] "Generic (PLEG): container finished" podID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerID="67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da" exitCode=0 Nov 02 15:44:57.407164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.407051 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" Nov 02 15:44:57.407164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.407082 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" event={"ID":"4c1aacac-a3f6-4b64-84ed-2c673e7edd59","Type":"ContainerDied","Data":"67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da"} Nov 02 15:44:57.407164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.407138 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7" event={"ID":"4c1aacac-a3f6-4b64-84ed-2c673e7edd59","Type":"ContainerDied","Data":"5027bf15518a36616c9c2b8f8b891625635efc956b4cd405aa6c3e2de2df1658"} Nov 02 15:44:57.407281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.407169 5433 scope.go:122] "RemoveContainer" containerID="67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da" Nov 02 15:44:57.416403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.416386 5433 scope.go:122] "RemoveContainer" containerID="978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c" Nov 02 15:44:57.425587 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.425570 5433 scope.go:122] "RemoveContainer" containerID="67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da" Nov 02 15:44:57.425902 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:44:57.425877 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da\": container with ID starting with 67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da not found: ID does not exist" containerID="67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da" Nov 02 15:44:57.426097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.425915 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da"} err="failed to get container status \"67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da\": rpc error: code = NotFound desc = could not find container \"67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da\": container with ID starting with 67ce71cee3c839933a55201a028741ea2839a1813f2c0e76ea7f7986857440da not found: ID does not exist" Nov 02 15:44:57.426097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.425939 5433 scope.go:122] "RemoveContainer" containerID="978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c" Nov 02 15:44:57.426315 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:44:57.426294 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c\": container with ID starting with 978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c not found: ID does not exist" containerID="978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c" Nov 02 15:44:57.426371 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.426332 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c"} err="failed to get container status \"978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c\": rpc error: code = NotFound desc = could not find container \"978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c\": container with ID starting with 978accdcf8d8a0f674faba9ee794bb6854467296bb4da7b286d2afd910a2045c not found: ID does not exist" Nov 02 15:44:57.561053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561013 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-credential-socket\") pod \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " Nov 02 15:44:57.561273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561122 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istiod-ca-cert\") pod \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " Nov 02 15:44:57.561273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561169 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-token\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-token\") pod \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " Nov 02 15:44:57.561273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561195 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-socket\") pod \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " Nov 02 15:44:57.561463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561339 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-credential-socket" pod "4c1aacac-a3f6-4b64-84ed-2c673e7edd59" (UID: "4c1aacac-a3f6-4b64-84ed-2c673e7edd59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:57.561463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561374 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-kube-api-access-77r8p\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-kube-api-access-77r8p\") pod \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " Nov 02 15:44:57.561581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561488 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-socket" pod "4c1aacac-a3f6-4b64-84ed-2c673e7edd59" (UID: "4c1aacac-a3f6-4b64-84ed-2c673e7edd59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:57.561581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561531 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-data\") pod \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " Nov 02 15:44:57.561581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561567 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istiod-ca-cert" pod "4c1aacac-a3f6-4b64-84ed-2c673e7edd59" (UID: "4c1aacac-a3f6-4b64-84ed-2c673e7edd59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:44:57.561749 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561572 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-podinfo\") pod \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " Nov 02 15:44:57.561805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561677 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-envoy\") pod \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " Nov 02 15:44:57.561864 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561795 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-data" pod "4c1aacac-a3f6-4b64-84ed-2c673e7edd59" (UID: "4c1aacac-a3f6-4b64-84ed-2c673e7edd59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:57.562132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.561855 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-certs\") pod \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\" (UID: \"4c1aacac-a3f6-4b64-84ed-2c673e7edd59\") " Nov 02 15:44:57.562219 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.562126 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-certs" pod "4c1aacac-a3f6-4b64-84ed-2c673e7edd59" (UID: "4c1aacac-a3f6-4b64-84ed-2c673e7edd59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:57.563031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.562909 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.563031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.562936 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.563031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.562952 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.563031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.562970 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.563031 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.563000 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.563725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.563703 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-podinfo" pod "4c1aacac-a3f6-4b64-84ed-2c673e7edd59" (UID: "4c1aacac-a3f6-4b64-84ed-2c673e7edd59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:44:57.563800 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.563779 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-token" pod "4c1aacac-a3f6-4b64-84ed-2c673e7edd59" (UID: "4c1aacac-a3f6-4b64-84ed-2c673e7edd59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:57.563881 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.563863 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-kube-api-access-77r8p" pod "4c1aacac-a3f6-4b64-84ed-2c673e7edd59" (UID: "4c1aacac-a3f6-4b64-84ed-2c673e7edd59"). InnerVolumeSpecName "kube-api-access-77r8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:57.563968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.563951 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-envoy" pod "4c1aacac-a3f6-4b64-84ed-2c673e7edd59" (UID: "4c1aacac-a3f6-4b64-84ed-2c673e7edd59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:57.664753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.664671 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.664753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.664700 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.664753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.664711 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.664753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.664721 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77r8p\" (UniqueName: \"kubernetes.io/projected/4c1aacac-a3f6-4b64-84ed-2c673e7edd59-kube-api-access-77r8p\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.829918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.829874 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7"] Nov 02 15:44:57.837784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:57.837753 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2c35ee-openshift-default-6bc744f999-4v9g7"] Nov 02 15:44:58.456849 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:44:58.456815 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" path="/var/lib/kubelet/pods/4c1aacac-a3f6-4b64-84ed-2c673e7edd59/volumes" Nov 02 15:45:00.852298 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:00.852255 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:04.607299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:04.607265 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw"] Nov 02 15:45:05.037568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:05.037533 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:05.204092 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:05.204053 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:06.278279 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:06.278229 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:09.607816 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.607764 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" podUID="bea5c62e-aa29-4f83-8ca9-abb02750fcfa" containerName="payload-processing" containerID="cri-o://9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17" gracePeriod=25 Nov 02 15:45:09.767180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767136 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4"] Nov 02 15:45:09.767775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767755 5433 cpu_manager.go:414] "containerMap: removing container" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" Nov 02 15:45:09.767775 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767773 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767790 5433 cpu_manager.go:414] "containerMap: removing container" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767795 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767803 5433 cpu_manager.go:414] "containerMap: removing container" podUID="95cc3a4b-68fe-42f3-b931-70b6d1132b05" containerName="payload-pre-processing" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767810 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cc3a4b-68fe-42f3-b931-70b6d1132b05" containerName="payload-pre-processing" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767828 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b54066c7-e857-4a05-a72c-f194d6a6b655" containerName="payload-processing" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767833 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54066c7-e857-4a05-a72c-f194d6a6b655" containerName="payload-processing" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767841 5433 cpu_manager.go:414] "containerMap: removing container" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767846 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767860 5433 cpu_manager.go:414] "containerMap: removing container" podUID="1193425a-4b6b-47ac-827f-20a00f789df1" containerName="maas-api" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767864 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1193425a-4b6b-47ac-827f-20a00f789df1" containerName="maas-api" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767881 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca" containerName="payload-processing" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767886 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca" containerName="payload-processing" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767893 5433 cpu_manager.go:414] "containerMap: removing container" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767898 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767985 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" Nov 02 15:45:09.767965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.767999 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="76eb87da-7c2e-4924-b606-3f97d446fc21" containerName="istio-proxy" Nov 02 15:45:09.768723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.768009 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cc3a4b-68fe-42f3-b931-70b6d1132b05" containerName="payload-pre-processing" Nov 02 15:45:09.768723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.768020 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54066c7-e857-4a05-a72c-f194d6a6b655" containerName="payload-processing" Nov 02 15:45:09.768723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.768026 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" Nov 02 15:45:09.768723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.768036 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="1193425a-4b6b-47ac-827f-20a00f789df1" containerName="maas-api" Nov 02 15:45:09.768723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.768043 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1aaa7bc-3c3a-43de-8c9f-c76311fa62ca" containerName="payload-processing" Nov 02 15:45:09.771541 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.771518 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:09.820852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.820819 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:45:09.823649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.823622 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38602001-6whp4\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:09.823649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.823663 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38602001-6whp4\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:09.823860 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.823800 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t48m\" (UniqueName: \"kubernetes.io/projected/12a57e1e-9341-4d1b-98a6-903cc883ecb2-kube-api-access-4t48m\") pod \"maas-api-revoke-sub-38602001-6whp4\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:09.866918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.866888 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:45:09.870726 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.870696 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4"] Nov 02 15:45:09.888191 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.888162 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:45:09.924777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.924751 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-plugins-config-volume\") pod \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\" (UID: \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\") " Nov 02 15:45:09.924961 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.924903 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-kube-api-access-m2ntj\" (UniqueName: \"kubernetes.io/projected/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-kube-api-access-m2ntj\") pod \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\" (UID: \"bea5c62e-aa29-4f83-8ca9-abb02750fcfa\") " Nov 02 15:45:09.925158 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.925139 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t48m\" (UniqueName: \"kubernetes.io/projected/12a57e1e-9341-4d1b-98a6-903cc883ecb2-kube-api-access-4t48m\") pod \"maas-api-revoke-sub-38602001-6whp4\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:09.925224 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.925165 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-plugins-config-volume" pod "bea5c62e-aa29-4f83-8ca9-abb02750fcfa" (UID: "bea5c62e-aa29-4f83-8ca9-abb02750fcfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:09.925367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.925349 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38602001-6whp4\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:09.925431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.925396 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38602001-6whp4\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:09.925519 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.925503 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:09.925956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.925936 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38602001-6whp4\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:09.926999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.926951 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-kube-api-access-m2ntj" pod "bea5c62e-aa29-4f83-8ca9-abb02750fcfa" (UID: "bea5c62e-aa29-4f83-8ca9-abb02750fcfa"). InnerVolumeSpecName "kube-api-access-m2ntj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:09.927661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.927643 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38602001-6whp4\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:09.982383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:09.982352 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t48m\" (UniqueName: \"kubernetes.io/projected/12a57e1e-9341-4d1b-98a6-903cc883ecb2-kube-api-access-4t48m\") pod \"maas-api-revoke-sub-38602001-6whp4\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:10.026818 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.026783 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2ntj\" (UniqueName: \"kubernetes.io/projected/bea5c62e-aa29-4f83-8ca9-abb02750fcfa-kube-api-access-m2ntj\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:10.091516 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.091483 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:10.460101 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.460074 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4"] Nov 02 15:45:10.463134 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:45:10.463106 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a57e1e_9341_4d1b_98a6_903cc883ecb2.slice/crio-85e817d1ce7a83cb858f3461b4b6fbc4d4e1188d78f145714f1d16a9040a8c9c WatchSource:0}: Error finding container 85e817d1ce7a83cb858f3461b4b6fbc4d4e1188d78f145714f1d16a9040a8c9c: Status 404 returned error can't find the container with id 85e817d1ce7a83cb858f3461b4b6fbc4d4e1188d78f145714f1d16a9040a8c9c Nov 02 15:45:10.497893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.497859 5433 generic.go:356] "Generic (PLEG): container finished" podID="bea5c62e-aa29-4f83-8ca9-abb02750fcfa" containerID="9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17" exitCode=0 Nov 02 15:45:10.498064 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.497910 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" event={"ID":"bea5c62e-aa29-4f83-8ca9-abb02750fcfa","Type":"ContainerDied","Data":"9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17"} Nov 02 15:45:10.498064 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.497917 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" Nov 02 15:45:10.498064 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.497966 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw" event={"ID":"bea5c62e-aa29-4f83-8ca9-abb02750fcfa","Type":"ContainerDied","Data":"0fa8d7d7b5cbfe3200b55305370c29e9ca6083d0cab3196c1c685ae2cf3e7257"} Nov 02 15:45:10.498242 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.498066 5433 scope.go:122] "RemoveContainer" containerID="9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17" Nov 02 15:45:10.499514 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.499433 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" event={"ID":"12a57e1e-9341-4d1b-98a6-903cc883ecb2","Type":"ContainerStarted","Data":"85e817d1ce7a83cb858f3461b4b6fbc4d4e1188d78f145714f1d16a9040a8c9c"} Nov 02 15:45:10.508776 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.508759 5433 scope.go:122] "RemoveContainer" containerID="9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17" Nov 02 15:45:10.509095 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:45:10.509074 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17\": container with ID starting with 9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17 not found: ID does not exist" containerID="9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17" Nov 02 15:45:10.509167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.509105 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17"} err="failed to get container status \"9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17\": rpc error: code = NotFound desc = could not find container \"9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17\": container with ID starting with 9bc9d22339a037c29dddb5688ffd3a73b4bc1c9d47ffb975c4c7dd5e6609da17 not found: ID does not exist" Nov 02 15:45:10.652098 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.652064 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw"] Nov 02 15:45:10.654630 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:10.654595 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3b351-5ffd97d8dc-llgrw"] Nov 02 15:45:11.012877 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:45:11.012842 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:45:11.492000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:11.491953 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:12.456744 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:12.456706 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bea5c62e-aa29-4f83-8ca9-abb02750fcfa" path="/var/lib/kubelet/pods/bea5c62e-aa29-4f83-8ca9-abb02750fcfa/volumes" Nov 02 15:45:16.543569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:16.543522 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:18.551573 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:18.551475 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" event={"ID":"12a57e1e-9341-4d1b-98a6-903cc883ecb2","Type":"ContainerStarted","Data":"72559dd67a03635e37cd77137a2b89804cbe81c97f80f617ee14257439ae3c1f"} Nov 02 15:45:18.661255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:18.661199 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" podStartSLOduration=2.23722969 podStartE2EDuration="9.661182951s" totalImagesPullingTime="7.423953261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:18.659823901 +0000 UTC m=+2822.333027119" watchObservedRunningTime="2026-11-02 15:45:18.661182951 +0000 UTC m=+2822.334386146" Nov 02 15:45:19.559820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:19.559783 5433 generic.go:356] "Generic (PLEG): container finished" podID="12a57e1e-9341-4d1b-98a6-903cc883ecb2" containerID="72559dd67a03635e37cd77137a2b89804cbe81c97f80f617ee14257439ae3c1f" exitCode=0 Nov 02 15:45:19.560263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:19.559818 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" event={"ID":"12a57e1e-9341-4d1b-98a6-903cc883ecb2","Type":"ContainerDied","Data":"72559dd67a03635e37cd77137a2b89804cbe81c97f80f617ee14257439ae3c1f"} Nov 02 15:45:20.732347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.732320 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:20.870637 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.870559 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-xks-ca\") pod \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " Nov 02 15:45:20.870637 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.870612 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a57e1e-9341-4d1b-98a6-903cc883ecb2-kube-api-access-4t48m\" (UniqueName: \"kubernetes.io/projected/12a57e1e-9341-4d1b-98a6-903cc883ecb2-kube-api-access-4t48m\") pod \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " Nov 02 15:45:20.870825 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.870765 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-openshift-ca\") pod \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\" (UID: \"12a57e1e-9341-4d1b-98a6-903cc883ecb2\") " Nov 02 15:45:20.871170 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.871139 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-openshift-ca" pod "12a57e1e-9341-4d1b-98a6-903cc883ecb2" (UID: "12a57e1e-9341-4d1b-98a6-903cc883ecb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:20.872904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.872875 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-xks-ca" pod "12a57e1e-9341-4d1b-98a6-903cc883ecb2" (UID: "12a57e1e-9341-4d1b-98a6-903cc883ecb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:20.873134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.873107 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a57e1e-9341-4d1b-98a6-903cc883ecb2-kube-api-access-4t48m" pod "12a57e1e-9341-4d1b-98a6-903cc883ecb2" (UID: "12a57e1e-9341-4d1b-98a6-903cc883ecb2"). InnerVolumeSpecName "kube-api-access-4t48m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:20.972375 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.972344 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-openshift-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:20.972375 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.972373 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a57e1e-9341-4d1b-98a6-903cc883ecb2-maas-api-xks-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:20.972375 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.972384 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t48m\" (UniqueName: \"kubernetes.io/projected/12a57e1e-9341-4d1b-98a6-903cc883ecb2-kube-api-access-4t48m\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:20.993936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.993907 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h"] Nov 02 15:45:20.994206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:20.994158 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" podUID="69375a37-c259-47f4-a4d7-f45a25a6e85b" containerName="maas-api" containerID="cri-o://092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab" gracePeriod=30 Nov 02 15:45:21.109958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.109922 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:21.273387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.273355 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:45:21.379657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.379626 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls\") pod \"69375a37-c259-47f4-a4d7-f45a25a6e85b\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " Nov 02 15:45:21.379657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.379662 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69375a37-c259-47f4-a4d7-f45a25a6e85b-kube-api-access-tkqdw\" (UniqueName: \"kubernetes.io/projected/69375a37-c259-47f4-a4d7-f45a25a6e85b-kube-api-access-tkqdw\") pod \"69375a37-c259-47f4-a4d7-f45a25a6e85b\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " Nov 02 15:45:21.379893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.379741 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69375a37-c259-47f4-a4d7-f45a25a6e85b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69375a37-c259-47f4-a4d7-f45a25a6e85b-openshift-service-ca\") pod \"69375a37-c259-47f4-a4d7-f45a25a6e85b\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " Nov 02 15:45:21.379893 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.379795 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-metrics-certs\") pod \"69375a37-c259-47f4-a4d7-f45a25a6e85b\" (UID: \"69375a37-c259-47f4-a4d7-f45a25a6e85b\") " Nov 02 15:45:21.380287 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.380245 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69375a37-c259-47f4-a4d7-f45a25a6e85b-openshift-service-ca" pod "69375a37-c259-47f4-a4d7-f45a25a6e85b" (UID: "69375a37-c259-47f4-a4d7-f45a25a6e85b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:21.382589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.382250 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls" pod "69375a37-c259-47f4-a4d7-f45a25a6e85b" (UID: "69375a37-c259-47f4-a4d7-f45a25a6e85b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:21.382589 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.382335 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-metrics-certs" pod "69375a37-c259-47f4-a4d7-f45a25a6e85b" (UID: "69375a37-c259-47f4-a4d7-f45a25a6e85b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:21.385389 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.385360 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69375a37-c259-47f4-a4d7-f45a25a6e85b-kube-api-access-tkqdw" pod "69375a37-c259-47f4-a4d7-f45a25a6e85b" (UID: "69375a37-c259-47f4-a4d7-f45a25a6e85b"). InnerVolumeSpecName "kube-api-access-tkqdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:21.481364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.481257 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69375a37-c259-47f4-a4d7-f45a25a6e85b-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.481364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.481286 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.481364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.481296 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69375a37-c259-47f4-a4d7-f45a25a6e85b-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.481364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.481304 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkqdw\" (UniqueName: \"kubernetes.io/projected/69375a37-c259-47f4-a4d7-f45a25a6e85b-kube-api-access-tkqdw\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.574350 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.574312 5433 generic.go:356] "Generic (PLEG): container finished" podID="69375a37-c259-47f4-a4d7-f45a25a6e85b" containerID="092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab" exitCode=0 Nov 02 15:45:21.574626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.574351 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" event={"ID":"69375a37-c259-47f4-a4d7-f45a25a6e85b","Type":"ContainerDied","Data":"092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab"} Nov 02 15:45:21.574626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.574387 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" Nov 02 15:45:21.574626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.574412 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h" event={"ID":"69375a37-c259-47f4-a4d7-f45a25a6e85b","Type":"ContainerDied","Data":"c4657c74a10938f6d9ceff56c785a3600ba6e1fb73fbe3d3e3f0a7ba8db954d9"} Nov 02 15:45:21.574626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.574440 5433 scope.go:122] "RemoveContainer" containerID="092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab" Nov 02 15:45:21.576338 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.576318 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" Nov 02 15:45:21.576487 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.576355 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4" event={"ID":"12a57e1e-9341-4d1b-98a6-903cc883ecb2","Type":"ContainerDied","Data":"85e817d1ce7a83cb858f3461b4b6fbc4d4e1188d78f145714f1d16a9040a8c9c"} Nov 02 15:45:21.576487 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.576395 5433 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e817d1ce7a83cb858f3461b4b6fbc4d4e1188d78f145714f1d16a9040a8c9c" Nov 02 15:45:21.587663 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.587645 5433 scope.go:122] "RemoveContainer" containerID="092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab" Nov 02 15:45:21.588211 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:45:21.588108 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab\": container with ID starting with 092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab not found: ID does not exist" containerID="092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab" Nov 02 15:45:21.588211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.588145 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab"} err="failed to get container status \"092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab\": rpc error: code = NotFound desc = could not find container \"092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab\": container with ID starting with 092c964654f0df9b896ad0ce501dfa9a78e41d55c842ceca90c6445edc7778ab not found: ID does not exist" Nov 02 15:45:21.738169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.738077 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h"] Nov 02 15:45:21.754685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:21.754651 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ca2cf-54d4bf6b59-pxt8h"] Nov 02 15:45:22.458284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:22.458250 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69375a37-c259-47f4-a4d7-f45a25a6e85b" path="/var/lib/kubelet/pods/69375a37-c259-47f4-a4d7-f45a25a6e85b/volumes" Nov 02 15:45:22.861527 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:22.861445 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:23.022347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:23.022307 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:23.181162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:23.174879 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:23.815840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:23.815807 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:25.469152 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:25.469115 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:28.613246 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:28.613213 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:30.498925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:30.498895 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:30.700444 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:30.700409 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:30.936037 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:30.936002 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:32.093029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:32.092993 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:32.241263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:32.241227 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:34.005833 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:34.005774 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:39.459614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:39.459541 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:41.011796 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:45:41.011759 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:45:43.343027 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:43.342969 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd"] Nov 02 15:45:43.650610 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:43.650569 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:43.732183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:43.732137 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86"] Nov 02 15:45:43.733598 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:43.733553 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" podUID="9f969b30-d040-4ae8-afcb-3b380e4b14fd" containerName="maas-api" containerID="cri-o://405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969" gracePeriod=30 Nov 02 15:45:43.797563 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:43.797524 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:43.998872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:43.998813 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:44.036149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.036119 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:45:44.189055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.186423 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-maas-api-tls\") pod \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " Nov 02 15:45:44.189055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.186512 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-metrics-certs\") pod \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " Nov 02 15:45:44.189055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.186651 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f969b30-d040-4ae8-afcb-3b380e4b14fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f969b30-d040-4ae8-afcb-3b380e4b14fd-openshift-service-ca\") pod \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " Nov 02 15:45:44.189055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.186741 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f969b30-d040-4ae8-afcb-3b380e4b14fd-kube-api-access-mtz56\" (UniqueName: \"kubernetes.io/projected/9f969b30-d040-4ae8-afcb-3b380e4b14fd-kube-api-access-mtz56\") pod \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\" (UID: \"9f969b30-d040-4ae8-afcb-3b380e4b14fd\") " Nov 02 15:45:44.201666 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.191798 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f969b30-d040-4ae8-afcb-3b380e4b14fd-openshift-service-ca" pod "9f969b30-d040-4ae8-afcb-3b380e4b14fd" (UID: "9f969b30-d040-4ae8-afcb-3b380e4b14fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:44.201666 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.197565 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-metrics-certs" pod "9f969b30-d040-4ae8-afcb-3b380e4b14fd" (UID: "9f969b30-d040-4ae8-afcb-3b380e4b14fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:44.201666 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.199104 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-maas-api-tls" pod "9f969b30-d040-4ae8-afcb-3b380e4b14fd" (UID: "9f969b30-d040-4ae8-afcb-3b380e4b14fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:44.204108 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.204079 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f969b30-d040-4ae8-afcb-3b380e4b14fd-kube-api-access-mtz56" pod "9f969b30-d040-4ae8-afcb-3b380e4b14fd" (UID: "9f969b30-d040-4ae8-afcb-3b380e4b14fd"). InnerVolumeSpecName "kube-api-access-mtz56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:44.319356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.303164 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:44.319356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.303213 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f969b30-d040-4ae8-afcb-3b380e4b14fd-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:44.319356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.303230 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtz56\" (UniqueName: \"kubernetes.io/projected/9f969b30-d040-4ae8-afcb-3b380e4b14fd-kube-api-access-mtz56\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:44.319356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.303245 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f969b30-d040-4ae8-afcb-3b380e4b14fd-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:44.732272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.732184 5433 generic.go:356] "Generic (PLEG): container finished" podID="9f969b30-d040-4ae8-afcb-3b380e4b14fd" containerID="405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969" exitCode=0 Nov 02 15:45:44.732818 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.732774 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" event={"ID":"9f969b30-d040-4ae8-afcb-3b380e4b14fd","Type":"ContainerDied","Data":"405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969"} Nov 02 15:45:44.733042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.733023 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" event={"ID":"9f969b30-d040-4ae8-afcb-3b380e4b14fd","Type":"ContainerDied","Data":"c1f08a4bb1bf8b75ce74819dda60d3754be58a0972b52345cdcdfceeaa8bd23d"} Nov 02 15:45:44.733190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.733176 5433 scope.go:122] "RemoveContainer" containerID="405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969" Nov 02 15:45:44.733486 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.733470 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86" Nov 02 15:45:44.743152 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.743133 5433 scope.go:122] "RemoveContainer" containerID="405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969" Nov 02 15:45:44.743537 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:45:44.743473 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969\": container with ID starting with 405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969 not found: ID does not exist" containerID="405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969" Nov 02 15:45:44.743537 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.743508 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969"} err="failed to get container status \"405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969\": rpc error: code = NotFound desc = could not find container \"405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969\": container with ID starting with 405cd50b69374c5e1346939a858ccfad0bbf70255d20b1840f4bca15b1e8d969 not found: ID does not exist" Nov 02 15:45:44.891800 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.891769 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86"] Nov 02 15:45:44.895337 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:44.895313 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb6bd4-6457b59d54-f7z86"] Nov 02 15:45:45.487519 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:45.487483 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:45.769180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:45.769074 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:45.914662 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:45.914623 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:46.456906 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:46.456872 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f969b30-d040-4ae8-afcb-3b380e4b14fd" path="/var/lib/kubelet/pods/9f969b30-d040-4ae8-afcb-3b380e4b14fd/volumes" Nov 02 15:45:46.458208 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:46.458187 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:48.344175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.344129 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" podUID="8eccaef4-511c-4792-8a2e-78fd5868d78a" containerName="payload-processing" containerID="cri-o://1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377" gracePeriod=25 Nov 02 15:45:48.608709 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.608641 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:45:48.757292 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.757255 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eccaef4-511c-4792-8a2e-78fd5868d78a-kube-api-access-f6gxc\" (UniqueName: \"kubernetes.io/projected/8eccaef4-511c-4792-8a2e-78fd5868d78a-kube-api-access-f6gxc\") pod \"8eccaef4-511c-4792-8a2e-78fd5868d78a\" (UID: \"8eccaef4-511c-4792-8a2e-78fd5868d78a\") " Nov 02 15:45:48.757507 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.757454 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eccaef4-511c-4792-8a2e-78fd5868d78a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eccaef4-511c-4792-8a2e-78fd5868d78a-plugins-config-volume\") pod \"8eccaef4-511c-4792-8a2e-78fd5868d78a\" (UID: \"8eccaef4-511c-4792-8a2e-78fd5868d78a\") " Nov 02 15:45:48.757863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.757826 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eccaef4-511c-4792-8a2e-78fd5868d78a-plugins-config-volume" pod "8eccaef4-511c-4792-8a2e-78fd5868d78a" (UID: "8eccaef4-511c-4792-8a2e-78fd5868d78a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:48.758109 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.758090 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eccaef4-511c-4792-8a2e-78fd5868d78a-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:48.759069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.759040 5433 generic.go:356] "Generic (PLEG): container finished" podID="8eccaef4-511c-4792-8a2e-78fd5868d78a" containerID="1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377" exitCode=0 Nov 02 15:45:48.759206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.759104 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" Nov 02 15:45:48.759206 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.759140 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" event={"ID":"8eccaef4-511c-4792-8a2e-78fd5868d78a","Type":"ContainerDied","Data":"1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377"} Nov 02 15:45:48.759333 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.759214 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd" event={"ID":"8eccaef4-511c-4792-8a2e-78fd5868d78a","Type":"ContainerDied","Data":"da1537e843149dc83adb1abd096126be0ce355ca57b3c7a8fafda3791c3051ee"} Nov 02 15:45:48.759333 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.759245 5433 scope.go:122] "RemoveContainer" containerID="1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377" Nov 02 15:45:48.759777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.759752 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eccaef4-511c-4792-8a2e-78fd5868d78a-kube-api-access-f6gxc" pod "8eccaef4-511c-4792-8a2e-78fd5868d78a" (UID: "8eccaef4-511c-4792-8a2e-78fd5868d78a"). InnerVolumeSpecName "kube-api-access-f6gxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:48.774924 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.774906 5433 scope.go:122] "RemoveContainer" containerID="1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377" Nov 02 15:45:48.775252 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:45:48.775233 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377\": container with ID starting with 1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377 not found: ID does not exist" containerID="1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377" Nov 02 15:45:48.775330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.775263 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377"} err="failed to get container status \"1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377\": rpc error: code = NotFound desc = could not find container \"1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377\": container with ID starting with 1d49763b239e70514be572dba689a992c1d2faa3201009fdea3d91330ed0a377 not found: ID does not exist" Nov 02 15:45:48.859324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:48.859244 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6gxc\" (UniqueName: \"kubernetes.io/projected/8eccaef4-511c-4792-8a2e-78fd5868d78a-kube-api-access-f6gxc\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:49.173230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:49.173197 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd"] Nov 02 15:45:49.182021 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:49.181990 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fe8ef1-59b4446495-482xd"] Nov 02 15:45:50.456132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:50.456098 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eccaef4-511c-4792-8a2e-78fd5868d78a" path="/var/lib/kubelet/pods/8eccaef4-511c-4792-8a2e-78fd5868d78a/volumes" Nov 02 15:45:51.255745 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:51.255705 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:52.956569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:52.956535 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:53.100029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:53.099994 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:53.278367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:53.278269 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:45:56.632792 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:45:56.632757 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:02.255425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:02.255385 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:05.627355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:05.627299 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp"] Nov 02 15:46:05.989428 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:05.989348 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:06.135253 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:06.135219 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:07.466468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:07.466429 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:10.628639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:10.628560 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" podUID="2fad0c18-a050-45bf-be6f-a030762dc6b5" containerName="payload-processing" containerID="cri-o://9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac" gracePeriod=25 Nov 02 15:46:10.896114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:10.896045 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:46:10.896261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:10.896107 5433 generic.go:356] "Generic (PLEG): container finished" podID="2fad0c18-a050-45bf-be6f-a030762dc6b5" containerID="9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac" exitCode=0 Nov 02 15:46:10.896330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:10.896254 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" event={"ID":"2fad0c18-a050-45bf-be6f-a030762dc6b5","Type":"ContainerDied","Data":"9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac"} Nov 02 15:46:10.896330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:10.896321 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" event={"ID":"2fad0c18-a050-45bf-be6f-a030762dc6b5","Type":"ContainerDied","Data":"eef698f63a1ae76b7789ba3143145185056e1404282f8879e2944bf1052bd41b"} Nov 02 15:46:10.896412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:10.896338 5433 scope.go:122] "RemoveContainer" containerID="9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac" Nov 02 15:46:10.905282 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:10.905264 5433 scope.go:122] "RemoveContainer" containerID="9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac" Nov 02 15:46:10.905566 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:10.905549 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac\": container with ID starting with 9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac not found: ID does not exist" containerID="9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac" Nov 02 15:46:10.905617 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:10.905574 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac"} err="failed to get container status \"9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac\": rpc error: code = NotFound desc = could not find container \"9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac\": container with ID starting with 9ff1a1a15cb2665d0a89dff744a73af76253d0592f7b009041d082e459e1bdac not found: ID does not exist" Nov 02 15:46:11.015072 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:11.015032 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:46:11.039688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:11.039653 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fad0c18-a050-45bf-be6f-a030762dc6b5-kube-api-access-psk44\" (UniqueName: \"kubernetes.io/projected/2fad0c18-a050-45bf-be6f-a030762dc6b5-kube-api-access-psk44\") pod \"2fad0c18-a050-45bf-be6f-a030762dc6b5\" (UID: \"2fad0c18-a050-45bf-be6f-a030762dc6b5\") " Nov 02 15:46:11.039863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:11.039701 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fad0c18-a050-45bf-be6f-a030762dc6b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fad0c18-a050-45bf-be6f-a030762dc6b5-plugins-config-volume\") pod \"2fad0c18-a050-45bf-be6f-a030762dc6b5\" (UID: \"2fad0c18-a050-45bf-be6f-a030762dc6b5\") " Nov 02 15:46:11.040181 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:11.040146 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fad0c18-a050-45bf-be6f-a030762dc6b5-plugins-config-volume" pod "2fad0c18-a050-45bf-be6f-a030762dc6b5" (UID: "2fad0c18-a050-45bf-be6f-a030762dc6b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:11.040356 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:11.040340 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fad0c18-a050-45bf-be6f-a030762dc6b5-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:11.041867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:11.041839 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fad0c18-a050-45bf-be6f-a030762dc6b5-kube-api-access-psk44" pod "2fad0c18-a050-45bf-be6f-a030762dc6b5" (UID: "2fad0c18-a050-45bf-be6f-a030762dc6b5"). InnerVolumeSpecName "kube-api-access-psk44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:11.141353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:11.141312 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psk44\" (UniqueName: \"kubernetes.io/projected/2fad0c18-a050-45bf-be6f-a030762dc6b5-kube-api-access-psk44\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:11.903127 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:11.903100 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp" Nov 02 15:46:12.058700 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:12.058663 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp"] Nov 02 15:46:12.076464 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:12.076433 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1ee37-ff74b8b9f-xsmkp"] Nov 02 15:46:12.456513 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:12.456482 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fad0c18-a050-45bf-be6f-a030762dc6b5" path="/var/lib/kubelet/pods/2fad0c18-a050-45bf-be6f-a030762dc6b5/volumes" Nov 02 15:46:12.552185 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:12.552131 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:17.845540 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:17.845500 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:22.652867 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:22.652832 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq"] Nov 02 15:46:22.653330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:22.653058 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" containerID="cri-o://7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906" gracePeriod=30 Nov 02 15:46:22.972661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:22.972565 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:24.015445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:24.015410 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:25.614450 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:25.614413 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb"] Nov 02 15:46:25.970091 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:25.970056 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:26.118784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:26.118744 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:27.911169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.911145 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:46:27.937072 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.936992 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435423ae-48db-4912-9d55-5bc14cbd7fbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435423ae-48db-4912-9d55-5bc14cbd7fbf-istiod-ca-cert\") pod \"435423ae-48db-4912-9d55-5bc14cbd7fbf\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " Nov 02 15:46:27.937072 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937037 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-credential-socket\") pod \"435423ae-48db-4912-9d55-5bc14cbd7fbf\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " Nov 02 15:46:27.937293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937118 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-envoy\") pod \"435423ae-48db-4912-9d55-5bc14cbd7fbf\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " Nov 02 15:46:27.937293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937184 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-certs\") pod \"435423ae-48db-4912-9d55-5bc14cbd7fbf\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " Nov 02 15:46:27.937293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937230 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-kube-api-access-kv6bx\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-kube-api-access-kv6bx\") pod \"435423ae-48db-4912-9d55-5bc14cbd7fbf\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " Nov 02 15:46:27.937293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937264 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-token\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-token\") pod \"435423ae-48db-4912-9d55-5bc14cbd7fbf\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " Nov 02 15:46:27.937501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937306 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-podinfo\") pod \"435423ae-48db-4912-9d55-5bc14cbd7fbf\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " Nov 02 15:46:27.937501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937365 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-socket\") pod \"435423ae-48db-4912-9d55-5bc14cbd7fbf\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " Nov 02 15:46:27.937501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937370 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435423ae-48db-4912-9d55-5bc14cbd7fbf-istiod-ca-cert" pod "435423ae-48db-4912-9d55-5bc14cbd7fbf" (UID: "435423ae-48db-4912-9d55-5bc14cbd7fbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:27.937501 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937445 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-data\") pod \"435423ae-48db-4912-9d55-5bc14cbd7fbf\" (UID: \"435423ae-48db-4912-9d55-5bc14cbd7fbf\") " Nov 02 15:46:27.937694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937503 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-certs" pod "435423ae-48db-4912-9d55-5bc14cbd7fbf" (UID: "435423ae-48db-4912-9d55-5bc14cbd7fbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:27.937753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937691 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-socket" pod "435423ae-48db-4912-9d55-5bc14cbd7fbf" (UID: "435423ae-48db-4912-9d55-5bc14cbd7fbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:27.937753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937725 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-credential-socket" pod "435423ae-48db-4912-9d55-5bc14cbd7fbf" (UID: "435423ae-48db-4912-9d55-5bc14cbd7fbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:27.937947 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.937923 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-data" pod "435423ae-48db-4912-9d55-5bc14cbd7fbf" (UID: "435423ae-48db-4912-9d55-5bc14cbd7fbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:27.938068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.938049 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:27.938113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.938075 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:27.938113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.938092 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:27.938113 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.938107 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435423ae-48db-4912-9d55-5bc14cbd7fbf-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:27.938214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.938122 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:27.939566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.939542 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-podinfo" pod "435423ae-48db-4912-9d55-5bc14cbd7fbf" (UID: "435423ae-48db-4912-9d55-5bc14cbd7fbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:46:27.939834 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.939804 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-envoy" pod "435423ae-48db-4912-9d55-5bc14cbd7fbf" (UID: "435423ae-48db-4912-9d55-5bc14cbd7fbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:27.939955 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.939885 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-kube-api-access-kv6bx" pod "435423ae-48db-4912-9d55-5bc14cbd7fbf" (UID: "435423ae-48db-4912-9d55-5bc14cbd7fbf"). InnerVolumeSpecName "kube-api-access-kv6bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:27.940130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.940021 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-token" pod "435423ae-48db-4912-9d55-5bc14cbd7fbf" (UID: "435423ae-48db-4912-9d55-5bc14cbd7fbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:27.960265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:27.960190 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:28.031027 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.030990 5433 generic.go:356] "Generic (PLEG): container finished" podID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerID="7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906" exitCode=0 Nov 02 15:46:28.031229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.031032 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" event={"ID":"435423ae-48db-4912-9d55-5bc14cbd7fbf","Type":"ContainerDied","Data":"7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906"} Nov 02 15:46:28.031229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.031051 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" Nov 02 15:46:28.031229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.031104 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq" event={"ID":"435423ae-48db-4912-9d55-5bc14cbd7fbf","Type":"ContainerDied","Data":"2376ad880814c685662bcbfa41d0351047d8e68a6d76e1c00b868995a0a92cad"} Nov 02 15:46:28.031229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.031133 5433 scope.go:122] "RemoveContainer" containerID="7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906" Nov 02 15:46:28.039326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.039214 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:28.039326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.039237 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv6bx\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-kube-api-access-kv6bx\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:28.039326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.039253 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:28.039326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.039262 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435423ae-48db-4912-9d55-5bc14cbd7fbf-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:28.042186 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.042167 5433 scope.go:122] "RemoveContainer" containerID="7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906" Nov 02 15:46:28.042476 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:28.042452 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906\": container with ID starting with 7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906 not found: ID does not exist" containerID="7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906" Nov 02 15:46:28.042581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.042480 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906"} err="failed to get container status \"7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906\": rpc error: code = NotFound desc = could not find container \"7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906\": container with ID starting with 7904bdf20af82e9d5c107cfb870a337d4752f1d98065ba9474f6e7dab152a906 not found: ID does not exist" Nov 02 15:46:28.186546 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.186511 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq"] Nov 02 15:46:28.194802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.194717 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-9dgcq"] Nov 02 15:46:28.456389 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:28.456312 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" path="/var/lib/kubelet/pods/435423ae-48db-4912-9d55-5bc14cbd7fbf/volumes" Nov 02 15:46:30.615002 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:30.614920 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" podUID="6fc8b9d9-3252-48b6-9644-3107f357b196" containerName="payload-processing" containerID="cri-o://560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3" gracePeriod=25 Nov 02 15:46:30.881856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:30.881825 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:46:30.974459 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:30.974428 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc8b9d9-3252-48b6-9644-3107f357b196-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8b9d9-3252-48b6-9644-3107f357b196-plugins-config-volume\") pod \"6fc8b9d9-3252-48b6-9644-3107f357b196\" (UID: \"6fc8b9d9-3252-48b6-9644-3107f357b196\") " Nov 02 15:46:30.974645 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:30.974550 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc8b9d9-3252-48b6-9644-3107f357b196-kube-api-access-dl4hc\" (UniqueName: \"kubernetes.io/projected/6fc8b9d9-3252-48b6-9644-3107f357b196-kube-api-access-dl4hc\") pod \"6fc8b9d9-3252-48b6-9644-3107f357b196\" (UID: \"6fc8b9d9-3252-48b6-9644-3107f357b196\") " Nov 02 15:46:30.974835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:30.974811 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc8b9d9-3252-48b6-9644-3107f357b196-plugins-config-volume" pod "6fc8b9d9-3252-48b6-9644-3107f357b196" (UID: "6fc8b9d9-3252-48b6-9644-3107f357b196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:30.975124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:30.975092 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc8b9d9-3252-48b6-9644-3107f357b196-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:30.976677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:30.976651 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc8b9d9-3252-48b6-9644-3107f357b196-kube-api-access-dl4hc" pod "6fc8b9d9-3252-48b6-9644-3107f357b196" (UID: "6fc8b9d9-3252-48b6-9644-3107f357b196"). InnerVolumeSpecName "kube-api-access-dl4hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:31.049552 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.049518 5433 generic.go:356] "Generic (PLEG): container finished" podID="6fc8b9d9-3252-48b6-9644-3107f357b196" containerID="560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3" exitCode=0 Nov 02 15:46:31.049552 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.049560 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" Nov 02 15:46:31.049763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.049596 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" event={"ID":"6fc8b9d9-3252-48b6-9644-3107f357b196","Type":"ContainerDied","Data":"560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3"} Nov 02 15:46:31.049763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.049649 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb" event={"ID":"6fc8b9d9-3252-48b6-9644-3107f357b196","Type":"ContainerDied","Data":"97a7fbafcbdbfcd0ced4922e45d415a9e32119eabca3d07e9137f0510749bd37"} Nov 02 15:46:31.049763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.049670 5433 scope.go:122] "RemoveContainer" containerID="560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3" Nov 02 15:46:31.059678 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.059657 5433 scope.go:122] "RemoveContainer" containerID="560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3" Nov 02 15:46:31.059935 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:31.059918 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3\": container with ID starting with 560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3 not found: ID does not exist" containerID="560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3" Nov 02 15:46:31.060016 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.059945 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3"} err="failed to get container status \"560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3\": rpc error: code = NotFound desc = could not find container \"560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3\": container with ID starting with 560e7afbbba8369b8f825b99231f7d2cb7de7bd3e483c35e4da76bb00f4b55f3 not found: ID does not exist" Nov 02 15:46:31.076628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.076604 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4hc\" (UniqueName: \"kubernetes.io/projected/6fc8b9d9-3252-48b6-9644-3107f357b196-kube-api-access-dl4hc\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:31.205073 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.204986 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb"] Nov 02 15:46:31.208780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:31.208749 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2203c8-5dcdfcb89-mxnwb"] Nov 02 15:46:32.457416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:32.457334 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc8b9d9-3252-48b6-9644-3107f357b196" path="/var/lib/kubelet/pods/6fc8b9d9-3252-48b6-9644-3107f357b196/volumes" Nov 02 15:46:32.486699 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:32.486661 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:33.718538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:33.718506 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:35.432463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:35.432421 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj"] Nov 02 15:46:35.433077 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:35.432657 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" containerID="cri-o://814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164" gracePeriod=30 Nov 02 15:46:35.479634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:35.479590 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9"] Nov 02 15:46:35.479844 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:35.479812 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" containerID="cri-o://8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf" gracePeriod=30 Nov 02 15:46:35.481908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:35.481873 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75"] Nov 02 15:46:35.482142 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:35.482107 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" containerID="cri-o://59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d" gracePeriod=30 Nov 02 15:46:35.526436 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:35.526033 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5"] Nov 02 15:46:35.526436 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:35.526246 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" containerID="cri-o://8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36" gracePeriod=30 Nov 02 15:46:38.071384 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:38.071334 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:40.764851 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.764819 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:46:40.855842 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.855810 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.885908 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-data\") pod \"48e5919a-b07b-4163-b8fa-910fd2352f06\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.885946 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-certs\") pod \"48e5919a-b07b-4163-b8fa-910fd2352f06\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.885969 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e5919a-b07b-4163-b8fa-910fd2352f06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e5919a-b07b-4163-b8fa-910fd2352f06-istiod-ca-cert\") pod \"48e5919a-b07b-4163-b8fa-910fd2352f06\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.886041 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-envoy\") pod \"48e5919a-b07b-4163-b8fa-910fd2352f06\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.886082 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-token\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-token\") pod \"48e5919a-b07b-4163-b8fa-910fd2352f06\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.886227 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-credential-socket\") pod \"48e5919a-b07b-4163-b8fa-910fd2352f06\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.886241 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-data" pod "48e5919a-b07b-4163-b8fa-910fd2352f06" (UID: "48e5919a-b07b-4163-b8fa-910fd2352f06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.886317 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-kube-api-access-khbxq\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-kube-api-access-khbxq\") pod \"48e5919a-b07b-4163-b8fa-910fd2352f06\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.886388 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-podinfo\") pod \"48e5919a-b07b-4163-b8fa-910fd2352f06\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " Nov 02 15:46:40.886600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.886431 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-socket\") pod \"48e5919a-b07b-4163-b8fa-910fd2352f06\" (UID: \"48e5919a-b07b-4163-b8fa-910fd2352f06\") " Nov 02 15:46:40.887296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.887031 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.887296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.887227 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-credential-socket" pod "48e5919a-b07b-4163-b8fa-910fd2352f06" (UID: "48e5919a-b07b-4163-b8fa-910fd2352f06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.887296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.887234 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-socket" pod "48e5919a-b07b-4163-b8fa-910fd2352f06" (UID: "48e5919a-b07b-4163-b8fa-910fd2352f06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.887494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.887471 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-certs" pod "48e5919a-b07b-4163-b8fa-910fd2352f06" (UID: "48e5919a-b07b-4163-b8fa-910fd2352f06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.887655 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.887623 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e5919a-b07b-4163-b8fa-910fd2352f06-istiod-ca-cert" pod "48e5919a-b07b-4163-b8fa-910fd2352f06" (UID: "48e5919a-b07b-4163-b8fa-910fd2352f06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:40.890131 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.890026 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-kube-api-access-khbxq" pod "48e5919a-b07b-4163-b8fa-910fd2352f06" (UID: "48e5919a-b07b-4163-b8fa-910fd2352f06"). InnerVolumeSpecName "kube-api-access-khbxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.890567 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.890507 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-token" pod "48e5919a-b07b-4163-b8fa-910fd2352f06" (UID: "48e5919a-b07b-4163-b8fa-910fd2352f06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.891140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.891114 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-podinfo" pod "48e5919a-b07b-4163-b8fa-910fd2352f06" (UID: "48e5919a-b07b-4163-b8fa-910fd2352f06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:46:40.893768 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.893746 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-envoy" pod "48e5919a-b07b-4163-b8fa-910fd2352f06" (UID: "48e5919a-b07b-4163-b8fa-910fd2352f06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.909519 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.909226 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:46:40.917446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.917329 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:46:40.987808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.987724 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-kube-api-access-ptpjj\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-kube-api-access-ptpjj\") pod \"3940e38e-34c7-49dd-b315-8a03373d0f59\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " Nov 02 15:46:40.987808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.987769 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-certs\") pod \"64d72143-9cba-4630-97c1-094c28c4e01c\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " Nov 02 15:46:40.987808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.987803 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-certs\") pod \"d54fee09-3c5a-4601-837d-5ca649fdbee9\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " Nov 02 15:46:40.988147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.987831 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3940e38e-34c7-49dd-b315-8a03373d0f59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3940e38e-34c7-49dd-b315-8a03373d0f59-istiod-ca-cert\") pod \"3940e38e-34c7-49dd-b315-8a03373d0f59\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " Nov 02 15:46:40.988147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.987869 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-envoy\") pod \"3940e38e-34c7-49dd-b315-8a03373d0f59\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " Nov 02 15:46:40.988147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.987908 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54fee09-3c5a-4601-837d-5ca649fdbee9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54fee09-3c5a-4601-837d-5ca649fdbee9-istiod-ca-cert\") pod \"d54fee09-3c5a-4601-837d-5ca649fdbee9\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " Nov 02 15:46:40.988147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.987943 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-credential-socket\") pod \"64d72143-9cba-4630-97c1-094c28c4e01c\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " Nov 02 15:46:40.988147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988001 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-podinfo\") pod \"3940e38e-34c7-49dd-b315-8a03373d0f59\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " Nov 02 15:46:40.988147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988044 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-data\") pod \"3940e38e-34c7-49dd-b315-8a03373d0f59\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " Nov 02 15:46:40.988147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988047 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-certs" pod "64d72143-9cba-4630-97c1-094c28c4e01c" (UID: "64d72143-9cba-4630-97c1-094c28c4e01c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.988147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988085 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-token\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-token\") pod \"3940e38e-34c7-49dd-b315-8a03373d0f59\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " Nov 02 15:46:40.988147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988120 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-podinfo\") pod \"d54fee09-3c5a-4601-837d-5ca649fdbee9\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988174 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-kube-api-access-8zxgd\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-kube-api-access-8zxgd\") pod \"64d72143-9cba-4630-97c1-094c28c4e01c\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988232 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-token\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-token\") pod \"d54fee09-3c5a-4601-837d-5ca649fdbee9\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988278 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-credential-socket\") pod \"d54fee09-3c5a-4601-837d-5ca649fdbee9\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988305 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-data\") pod \"d54fee09-3c5a-4601-837d-5ca649fdbee9\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988304 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-certs" pod "d54fee09-3c5a-4601-837d-5ca649fdbee9" (UID: "d54fee09-3c5a-4601-837d-5ca649fdbee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988345 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-kube-api-access-4kw2t\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-kube-api-access-4kw2t\") pod \"d54fee09-3c5a-4601-837d-5ca649fdbee9\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988348 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3940e38e-34c7-49dd-b315-8a03373d0f59-istiod-ca-cert" pod "3940e38e-34c7-49dd-b315-8a03373d0f59" (UID: "3940e38e-34c7-49dd-b315-8a03373d0f59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988377 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64d72143-9cba-4630-97c1-094c28c4e01c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d72143-9cba-4630-97c1-094c28c4e01c-istio-podinfo\") pod \"64d72143-9cba-4630-97c1-094c28c4e01c\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988403 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54fee09-3c5a-4601-837d-5ca649fdbee9-istiod-ca-cert" pod "d54fee09-3c5a-4601-837d-5ca649fdbee9" (UID: "d54fee09-3c5a-4601-837d-5ca649fdbee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988446 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-socket\") pod \"64d72143-9cba-4630-97c1-094c28c4e01c\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988476 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-envoy\") pod \"d54fee09-3c5a-4601-837d-5ca649fdbee9\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988510 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-socket\") pod \"3940e38e-34c7-49dd-b315-8a03373d0f59\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988559 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-data\") pod \"64d72143-9cba-4630-97c1-094c28c4e01c\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988584 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-credential-socket" pod "d54fee09-3c5a-4601-837d-5ca649fdbee9" (UID: "d54fee09-3c5a-4601-837d-5ca649fdbee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988608 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-socket\") pod \"d54fee09-3c5a-4601-837d-5ca649fdbee9\" (UID: \"d54fee09-3c5a-4601-837d-5ca649fdbee9\") " Nov 02 15:46:40.988657 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988643 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-istio-token\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-istio-token\") pod \"64d72143-9cba-4630-97c1-094c28c4e01c\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988669 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-data" pod "d54fee09-3c5a-4601-837d-5ca649fdbee9" (UID: "d54fee09-3c5a-4601-837d-5ca649fdbee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988677 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-envoy\") pod \"64d72143-9cba-4630-97c1-094c28c4e01c\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988715 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-credential-socket\") pod \"3940e38e-34c7-49dd-b315-8a03373d0f59\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988763 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d72143-9cba-4630-97c1-094c28c4e01c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d72143-9cba-4630-97c1-094c28c4e01c-istiod-ca-cert\") pod \"64d72143-9cba-4630-97c1-094c28c4e01c\" (UID: \"64d72143-9cba-4630-97c1-094c28c4e01c\") " Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.988796 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-certs\") pod \"3940e38e-34c7-49dd-b315-8a03373d0f59\" (UID: \"3940e38e-34c7-49dd-b315-8a03373d0f59\") " Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989154 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-data" pod "3940e38e-34c7-49dd-b315-8a03373d0f59" (UID: "3940e38e-34c7-49dd-b315-8a03373d0f59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989246 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-credential-socket" pod "64d72143-9cba-4630-97c1-094c28c4e01c" (UID: "64d72143-9cba-4630-97c1-094c28c4e01c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989285 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-data" pod "64d72143-9cba-4630-97c1-094c28c4e01c" (UID: "64d72143-9cba-4630-97c1-094c28c4e01c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989379 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-certs" pod "3940e38e-34c7-49dd-b315-8a03373d0f59" (UID: "3940e38e-34c7-49dd-b315-8a03373d0f59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.989463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989462 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989482 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989499 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989515 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989529 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989543 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3940e38e-34c7-49dd-b315-8a03373d0f59-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989558 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khbxq\" (UniqueName: \"kubernetes.io/projected/48e5919a-b07b-4163-b8fa-910fd2352f06-kube-api-access-khbxq\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989574 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54fee09-3c5a-4601-837d-5ca649fdbee9-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989576 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-credential-socket" pod "3940e38e-34c7-49dd-b315-8a03373d0f59" (UID: "3940e38e-34c7-49dd-b315-8a03373d0f59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989589 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989627 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989649 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989665 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989681 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989689 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d72143-9cba-4630-97c1-094c28c4e01c-istiod-ca-cert" pod "64d72143-9cba-4630-97c1-094c28c4e01c" (UID: "64d72143-9cba-4630-97c1-094c28c4e01c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989696 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e5919a-b07b-4163-b8fa-910fd2352f06-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989728 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e5919a-b07b-4163-b8fa-910fd2352f06-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989746 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989761 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.990018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.989965 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-socket" pod "64d72143-9cba-4630-97c1-094c28c4e01c" (UID: "64d72143-9cba-4630-97c1-094c28c4e01c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.991079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.990536 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-socket" pod "3940e38e-34c7-49dd-b315-8a03373d0f59" (UID: "3940e38e-34c7-49dd-b315-8a03373d0f59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.991079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.990598 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-envoy" pod "3940e38e-34c7-49dd-b315-8a03373d0f59" (UID: "3940e38e-34c7-49dd-b315-8a03373d0f59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.991079 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.990815 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-socket" pod "d54fee09-3c5a-4601-837d-5ca649fdbee9" (UID: "d54fee09-3c5a-4601-837d-5ca649fdbee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.991395 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.991355 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-kube-api-access-ptpjj" pod "3940e38e-34c7-49dd-b315-8a03373d0f59" (UID: "3940e38e-34c7-49dd-b315-8a03373d0f59"). InnerVolumeSpecName "kube-api-access-ptpjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.992219 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.992149 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-kube-api-access-8zxgd" pod "64d72143-9cba-4630-97c1-094c28c4e01c" (UID: "64d72143-9cba-4630-97c1-094c28c4e01c"). InnerVolumeSpecName "kube-api-access-8zxgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.992311 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.992230 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-podinfo" pod "d54fee09-3c5a-4601-837d-5ca649fdbee9" (UID: "d54fee09-3c5a-4601-837d-5ca649fdbee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:46:40.994222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.994194 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-envoy" pod "64d72143-9cba-4630-97c1-094c28c4e01c" (UID: "64d72143-9cba-4630-97c1-094c28c4e01c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.994222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.994211 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-envoy" pod "d54fee09-3c5a-4601-837d-5ca649fdbee9" (UID: "d54fee09-3c5a-4601-837d-5ca649fdbee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.994346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.994216 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-token" pod "d54fee09-3c5a-4601-837d-5ca649fdbee9" (UID: "d54fee09-3c5a-4601-837d-5ca649fdbee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.994346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.994228 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64d72143-9cba-4630-97c1-094c28c4e01c-istio-podinfo" pod "64d72143-9cba-4630-97c1-094c28c4e01c" (UID: "64d72143-9cba-4630-97c1-094c28c4e01c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:46:40.994346 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.994261 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-istio-token" pod "64d72143-9cba-4630-97c1-094c28c4e01c" (UID: "64d72143-9cba-4630-97c1-094c28c4e01c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.994769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.994739 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-token" pod "3940e38e-34c7-49dd-b315-8a03373d0f59" (UID: "3940e38e-34c7-49dd-b315-8a03373d0f59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.994853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.994831 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-kube-api-access-4kw2t" pod "d54fee09-3c5a-4601-837d-5ca649fdbee9" (UID: "d54fee09-3c5a-4601-837d-5ca649fdbee9"). InnerVolumeSpecName "kube-api-access-4kw2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.995371 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:40.995343 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-podinfo" pod "3940e38e-34c7-49dd-b315-8a03373d0f59" (UID: "3940e38e-34c7-49dd-b315-8a03373d0f59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:46:41.013656 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:41.013625 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:46:41.090916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.090879 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.090916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.090910 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zxgd\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-kube-api-access-8zxgd\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.090916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.090921 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.090930 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kw2t\" (UniqueName: \"kubernetes.io/projected/d54fee09-3c5a-4601-837d-5ca649fdbee9-kube-api-access-4kw2t\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.090940 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d72143-9cba-4630-97c1-094c28c4e01c-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.090949 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.090959 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.090968 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091002 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54fee09-3c5a-4601-837d-5ca649fdbee9-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091015 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d72143-9cba-4630-97c1-094c28c4e01c-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091026 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d72143-9cba-4630-97c1-094c28c4e01c-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091034 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091042 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d72143-9cba-4630-97c1-094c28c4e01c-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091050 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091060 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptpjj\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-kube-api-access-ptpjj\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091069 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091078 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.091179 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.091086 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3940e38e-34c7-49dd-b315-8a03373d0f59-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.113928 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.113894 5433 generic.go:356] "Generic (PLEG): container finished" podID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerID="814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164" exitCode=0 Nov 02 15:46:41.114097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.113941 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" Nov 02 15:46:41.114097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.113997 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" event={"ID":"48e5919a-b07b-4163-b8fa-910fd2352f06","Type":"ContainerDied","Data":"814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164"} Nov 02 15:46:41.114097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.114055 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj" event={"ID":"48e5919a-b07b-4163-b8fa-910fd2352f06","Type":"ContainerDied","Data":"89687ab38cb93cb8bc2703e4f6ff1de22374b75c1b06d82d2b7c7f41675d5a69"} Nov 02 15:46:41.114097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.114081 5433 scope.go:122] "RemoveContainer" containerID="814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164" Nov 02 15:46:41.115468 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.115435 5433 generic.go:356] "Generic (PLEG): container finished" podID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerID="59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d" exitCode=0 Nov 02 15:46:41.115600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.115484 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" Nov 02 15:46:41.115600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.115491 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" event={"ID":"3940e38e-34c7-49dd-b315-8a03373d0f59","Type":"ContainerDied","Data":"59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d"} Nov 02 15:46:41.115600 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.115538 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75" event={"ID":"3940e38e-34c7-49dd-b315-8a03373d0f59","Type":"ContainerDied","Data":"16c821e866c35d15acbbb21d2df7de1c1e182047abb0d58c6e42105aaf1b1558"} Nov 02 15:46:41.122440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.122414 5433 generic.go:356] "Generic (PLEG): container finished" podID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerID="8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36" exitCode=0 Nov 02 15:46:41.122685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.122458 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" Nov 02 15:46:41.122685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.122492 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" event={"ID":"d54fee09-3c5a-4601-837d-5ca649fdbee9","Type":"ContainerDied","Data":"8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36"} Nov 02 15:46:41.122685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.122540 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5" event={"ID":"d54fee09-3c5a-4601-837d-5ca649fdbee9","Type":"ContainerDied","Data":"ae520c517e7eb1ad88bf2004e1daa8d40dea6f4f41d14d9e7c7e1e9e4ce60baa"} Nov 02 15:46:41.124084 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.124058 5433 generic.go:356] "Generic (PLEG): container finished" podID="64d72143-9cba-4630-97c1-094c28c4e01c" containerID="8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf" exitCode=0 Nov 02 15:46:41.124202 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.124101 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" event={"ID":"64d72143-9cba-4630-97c1-094c28c4e01c","Type":"ContainerDied","Data":"8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf"} Nov 02 15:46:41.124202 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.124140 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" event={"ID":"64d72143-9cba-4630-97c1-094c28c4e01c","Type":"ContainerDied","Data":"da8acb53a91208ad8dc7e5b0103e55fd9c3190e9e863c18cea02a7db3bdea9b0"} Nov 02 15:46:41.124315 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.124211 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9" Nov 02 15:46:41.127167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.127147 5433 scope.go:122] "RemoveContainer" containerID="814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164" Nov 02 15:46:41.127505 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:41.127478 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164\": container with ID starting with 814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164 not found: ID does not exist" containerID="814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164" Nov 02 15:46:41.127628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.127519 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164"} err="failed to get container status \"814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164\": rpc error: code = NotFound desc = could not find container \"814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164\": container with ID starting with 814f2adcf9b0395bc163abb136971c706205b5dc835615fbd52bbdc4f5af1164 not found: ID does not exist" Nov 02 15:46:41.127628 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.127543 5433 scope.go:122] "RemoveContainer" containerID="59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d" Nov 02 15:46:41.137626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.137608 5433 scope.go:122] "RemoveContainer" containerID="59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d" Nov 02 15:46:41.137880 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:41.137862 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d\": container with ID starting with 59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d not found: ID does not exist" containerID="59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d" Nov 02 15:46:41.137925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.137889 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d"} err="failed to get container status \"59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d\": rpc error: code = NotFound desc = could not find container \"59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d\": container with ID starting with 59daac20765926964967745ec74747f048555fba58b321cb73542ca626d6a84d not found: ID does not exist" Nov 02 15:46:41.137925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.137908 5433 scope.go:122] "RemoveContainer" containerID="8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36" Nov 02 15:46:41.146858 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.146837 5433 scope.go:122] "RemoveContainer" containerID="8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36" Nov 02 15:46:41.147150 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:41.147128 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36\": container with ID starting with 8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36 not found: ID does not exist" containerID="8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36" Nov 02 15:46:41.147229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.147156 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36"} err="failed to get container status \"8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36\": rpc error: code = NotFound desc = could not find container \"8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36\": container with ID starting with 8fdec8d9991ebe983205978637f74505dced04308b8b4c6f6c10a11bc1a7ff36 not found: ID does not exist" Nov 02 15:46:41.147229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.147172 5433 scope.go:122] "RemoveContainer" containerID="8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf" Nov 02 15:46:41.156445 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.156426 5433 scope.go:122] "RemoveContainer" containerID="8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf" Nov 02 15:46:41.156900 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:41.156771 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf\": container with ID starting with 8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf not found: ID does not exist" containerID="8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf" Nov 02 15:46:41.157020 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.156908 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf"} err="failed to get container status \"8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf\": rpc error: code = NotFound desc = could not find container \"8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf\": container with ID starting with 8226e5f5b5125254f8cfe08707ab845ad3349dea1d8cd2694a38ef9510120cbf not found: ID does not exist" Nov 02 15:46:41.384332 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.384301 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75"] Nov 02 15:46:41.388370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.388342 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2vz75"] Nov 02 15:46:41.492852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.492818 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5"] Nov 02 15:46:41.498418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.498385 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-qgfx5"] Nov 02 15:46:41.601147 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.601112 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9"] Nov 02 15:46:41.607703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.607677 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-92bt9"] Nov 02 15:46:41.711638 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.711557 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj"] Nov 02 15:46:41.716681 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:41.716645 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x2fcj"] Nov 02 15:46:42.456200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:42.456168 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" path="/var/lib/kubelet/pods/3940e38e-34c7-49dd-b315-8a03373d0f59/volumes" Nov 02 15:46:42.456642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:42.456628 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" path="/var/lib/kubelet/pods/48e5919a-b07b-4163-b8fa-910fd2352f06/volumes" Nov 02 15:46:42.457042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:42.457029 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" path="/var/lib/kubelet/pods/64d72143-9cba-4630-97c1-094c28c4e01c/volumes" Nov 02 15:46:42.457753 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:42.457739 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" path="/var/lib/kubelet/pods/d54fee09-3c5a-4601-837d-5ca649fdbee9/volumes" Nov 02 15:46:43.164422 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:43.164384 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:47.440508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.440471 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw"] Nov 02 15:46:47.440946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.440668 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" podUID="c4a51c2d-c28e-4145-91ca-2e3f04f8101c" containerName="maas-api" containerID="cri-o://b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836" gracePeriod=30 Nov 02 15:46:47.546305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.546267 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:47.706607 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.706576 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:46:47.765276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.765247 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-openshift-service-ca\") pod \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " Nov 02 15:46:47.765441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.765298 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-kube-api-access-z9nwk\" (UniqueName: \"kubernetes.io/projected/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-kube-api-access-z9nwk\") pod \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " Nov 02 15:46:47.765441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.765327 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls\") pod \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " Nov 02 15:46:47.765441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.765383 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-metrics-certs\") pod \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\" (UID: \"c4a51c2d-c28e-4145-91ca-2e3f04f8101c\") " Nov 02 15:46:47.765626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.765589 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-openshift-service-ca" pod "c4a51c2d-c28e-4145-91ca-2e3f04f8101c" (UID: "c4a51c2d-c28e-4145-91ca-2e3f04f8101c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:47.765818 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.765791 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:47.767495 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.767469 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls" pod "c4a51c2d-c28e-4145-91ca-2e3f04f8101c" (UID: "c4a51c2d-c28e-4145-91ca-2e3f04f8101c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:46:47.767606 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.767545 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-kube-api-access-z9nwk" pod "c4a51c2d-c28e-4145-91ca-2e3f04f8101c" (UID: "c4a51c2d-c28e-4145-91ca-2e3f04f8101c"). InnerVolumeSpecName "kube-api-access-z9nwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:47.767606 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.767552 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-metrics-certs" pod "c4a51c2d-c28e-4145-91ca-2e3f04f8101c" (UID: "c4a51c2d-c28e-4145-91ca-2e3f04f8101c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:46:47.866945 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.866906 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9nwk\" (UniqueName: \"kubernetes.io/projected/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-kube-api-access-z9nwk\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:47.866945 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.866945 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:47.867219 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:47.866962 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4a51c2d-c28e-4145-91ca-2e3f04f8101c-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:48.169919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.169883 5433 generic.go:356] "Generic (PLEG): container finished" podID="c4a51c2d-c28e-4145-91ca-2e3f04f8101c" containerID="b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836" exitCode=0 Nov 02 15:46:48.170155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.169933 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" Nov 02 15:46:48.170155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.169997 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" event={"ID":"c4a51c2d-c28e-4145-91ca-2e3f04f8101c","Type":"ContainerDied","Data":"b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836"} Nov 02 15:46:48.170155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.170053 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw" event={"ID":"c4a51c2d-c28e-4145-91ca-2e3f04f8101c","Type":"ContainerDied","Data":"a53b7108a8e83081d359f897c927000a75e74bf22020888d07b79a510ed5b6ce"} Nov 02 15:46:48.170155 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.170083 5433 scope.go:122] "RemoveContainer" containerID="b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836" Nov 02 15:46:48.179457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.179439 5433 scope.go:122] "RemoveContainer" containerID="b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836" Nov 02 15:46:48.179697 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:46:48.179681 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836\": container with ID starting with b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836 not found: ID does not exist" containerID="b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836" Nov 02 15:46:48.179766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.179703 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836"} err="failed to get container status \"b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836\": rpc error: code = NotFound desc = could not find container \"b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836\": container with ID starting with b3778375369bc8482a938d3a3bd98ed08a2283098e4bf4ebb2bba0403090b836 not found: ID does not exist" Nov 02 15:46:48.327721 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.327691 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw"] Nov 02 15:46:48.374454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.374420 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e3b351-8496ff996c-v69sw"] Nov 02 15:46:48.457162 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:48.457088 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a51c2d-c28e-4145-91ca-2e3f04f8101c" path="/var/lib/kubelet/pods/c4a51c2d-c28e-4145-91ca-2e3f04f8101c/volumes" Nov 02 15:46:49.128327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:49.128277 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:49.365231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:49.365197 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:49.515419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:49.515331 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:50.097170 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:50.097132 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:54.698482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:54.698444 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:56.136729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:56.136691 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:56.284289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:56.284253 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:56.504511 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:56.504429 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:57.091682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:57.091647 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:46:59.948866 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:46:59.948831 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:02.051990 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:02.051942 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl"] Nov 02 15:47:02.098702 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:02.098668 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7"] Nov 02 15:47:02.151538 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:02.151500 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:05.406752 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:05.406720 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:09.104470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:09.104434 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r"] Nov 02 15:47:09.209140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:09.209107 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk"] Nov 02 15:47:09.461919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:09.461878 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:09.600877 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:09.600838 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:09.771065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:09.770960 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24"] Nov 02 15:47:09.771229 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:09.771200 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" podUID="07fa338b-7a5f-42de-b42b-8037d94048d9" containerName="maas-api" containerID="cri-o://45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238" gracePeriod=30 Nov 02 15:47:09.875277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:09.875239 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:10.048053 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.046708 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:47:10.108077 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.108029 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07fa338b-7a5f-42de-b42b-8037d94048d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07fa338b-7a5f-42de-b42b-8037d94048d9-openshift-service-ca\") pod \"07fa338b-7a5f-42de-b42b-8037d94048d9\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " Nov 02 15:47:10.108583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.108144 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fa338b-7a5f-42de-b42b-8037d94048d9-kube-api-access-s67mq\" (UniqueName: \"kubernetes.io/projected/07fa338b-7a5f-42de-b42b-8037d94048d9-kube-api-access-s67mq\") pod \"07fa338b-7a5f-42de-b42b-8037d94048d9\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " Nov 02 15:47:10.108583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.108202 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-metrics-certs\") pod \"07fa338b-7a5f-42de-b42b-8037d94048d9\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " Nov 02 15:47:10.108583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.108244 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls\") pod \"07fa338b-7a5f-42de-b42b-8037d94048d9\" (UID: \"07fa338b-7a5f-42de-b42b-8037d94048d9\") " Nov 02 15:47:10.108583 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.108477 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fa338b-7a5f-42de-b42b-8037d94048d9-openshift-service-ca" pod "07fa338b-7a5f-42de-b42b-8037d94048d9" (UID: "07fa338b-7a5f-42de-b42b-8037d94048d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:10.108838 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.108632 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07fa338b-7a5f-42de-b42b-8037d94048d9-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:10.110390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.110361 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls" pod "07fa338b-7a5f-42de-b42b-8037d94048d9" (UID: "07fa338b-7a5f-42de-b42b-8037d94048d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:10.110514 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.110415 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-metrics-certs" pod "07fa338b-7a5f-42de-b42b-8037d94048d9" (UID: "07fa338b-7a5f-42de-b42b-8037d94048d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:10.110514 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.110432 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fa338b-7a5f-42de-b42b-8037d94048d9-kube-api-access-s67mq" pod "07fa338b-7a5f-42de-b42b-8037d94048d9" (UID: "07fa338b-7a5f-42de-b42b-8037d94048d9"). InnerVolumeSpecName "kube-api-access-s67mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:10.209728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.209690 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s67mq\" (UniqueName: \"kubernetes.io/projected/07fa338b-7a5f-42de-b42b-8037d94048d9-kube-api-access-s67mq\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:10.209728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.209723 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:10.209728 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.209735 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07fa338b-7a5f-42de-b42b-8037d94048d9-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:10.296581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.296545 5433 generic.go:356] "Generic (PLEG): container finished" podID="07fa338b-7a5f-42de-b42b-8037d94048d9" containerID="45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238" exitCode=0 Nov 02 15:47:10.296774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.296595 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" event={"ID":"07fa338b-7a5f-42de-b42b-8037d94048d9","Type":"ContainerDied","Data":"45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238"} Nov 02 15:47:10.296774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.296603 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" Nov 02 15:47:10.296774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.296651 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24" event={"ID":"07fa338b-7a5f-42de-b42b-8037d94048d9","Type":"ContainerDied","Data":"f76838d56cb7e59fc48c463aa97f216b999303b4e2a5d307454fffed3092779f"} Nov 02 15:47:10.296774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.296674 5433 scope.go:122] "RemoveContainer" containerID="45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238" Nov 02 15:47:10.306305 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.306281 5433 scope.go:122] "RemoveContainer" containerID="45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238" Nov 02 15:47:10.306595 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:10.306575 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238\": container with ID starting with 45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238 not found: ID does not exist" containerID="45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238" Nov 02 15:47:10.306648 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.306603 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238"} err="failed to get container status \"45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238\": rpc error: code = NotFound desc = could not find container \"45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238\": container with ID starting with 45a565ec78bb194e8adcfd9ad76c30b126c5aae14aeec877d7b1d4423f3b2238 not found: ID does not exist" Nov 02 15:47:10.456390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.456360 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24"] Nov 02 15:47:10.458944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:10.458918 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2203c8-6ff69b6b9d-mgj24"] Nov 02 15:47:11.011324 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:11.011285 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:47:11.517871 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:11.517840 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:11.758222 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:11.758180 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:11.902491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:11.902456 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:12.456391 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:12.456354 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fa338b-7a5f-42de-b42b-8037d94048d9" path="/var/lib/kubelet/pods/07fa338b-7a5f-42de-b42b-8037d94048d9/volumes" Nov 02 15:47:12.462275 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:12.462253 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:14.105029 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:14.104950 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" podUID="500c39b4-df23-4fe6-abeb-46b7b9172de2" containerName="payload-processing" containerID="cri-o://ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444" gracePeriod=25 Nov 02 15:47:14.209925 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:14.209865 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" podUID="373b0af7-e073-474c-b465-8c0b22b359af" containerName="payload-pre-processing" containerID="cri-o://4d793d05a38e8a1efa198326dda70bcecc3d3f47e35ea023f94c23e9f40353d6" gracePeriod=25 Nov 02 15:47:14.264250 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:14.264208 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:14.628546 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:14.628514 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" podUID="500c39b4-df23-4fe6-abeb-46b7b9172de2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:16.539529 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.539490 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv"] Nov 02 15:47:16.539956 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.539708 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" podUID="440458b0-3f93-4f8a-a9b8-8f0e321bc22c" containerName="maas-api" containerID="cri-o://724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa" gracePeriod=30 Nov 02 15:47:16.655510 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.655472 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:16.805069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.805009 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:47:16.880613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.880583 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-metrics-certs\") pod \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " Nov 02 15:47:16.880774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.880627 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-openshift-service-ca\") pod \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " Nov 02 15:47:16.880774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.880744 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls\") pod \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " Nov 02 15:47:16.880863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.880786 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-kube-api-access-2kf7z\" (UniqueName: \"kubernetes.io/projected/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-kube-api-access-2kf7z\") pod \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\" (UID: \"440458b0-3f93-4f8a-a9b8-8f0e321bc22c\") " Nov 02 15:47:16.881080 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.881051 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-openshift-service-ca" pod "440458b0-3f93-4f8a-a9b8-8f0e321bc22c" (UID: "440458b0-3f93-4f8a-a9b8-8f0e321bc22c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:16.882840 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.882812 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-metrics-certs" pod "440458b0-3f93-4f8a-a9b8-8f0e321bc22c" (UID: "440458b0-3f93-4f8a-a9b8-8f0e321bc22c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:16.882940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.882851 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-kube-api-access-2kf7z" pod "440458b0-3f93-4f8a-a9b8-8f0e321bc22c" (UID: "440458b0-3f93-4f8a-a9b8-8f0e321bc22c"). InnerVolumeSpecName "kube-api-access-2kf7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:16.882940 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.882913 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls" pod "440458b0-3f93-4f8a-a9b8-8f0e321bc22c" (UID: "440458b0-3f93-4f8a-a9b8-8f0e321bc22c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:16.981873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.981842 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:16.981873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.981870 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:16.981873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.981880 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:16.982133 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:16.981891 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kf7z\" (UniqueName: \"kubernetes.io/projected/440458b0-3f93-4f8a-a9b8-8f0e321bc22c-kube-api-access-2kf7z\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.073549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.073457 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" containerID="cri-o://260cbda54fd6ca76b62537ce32e3533da036eb7f5a2f9ed96e06088e1302c29a" gracePeriod=45 Nov 02 15:47:17.119314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.119259 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" containerID="cri-o://82562aff686ef6fdeeeedb44c0c6702a68b33fcd98c21906e729670348ce7510" gracePeriod=45 Nov 02 15:47:17.273183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.273138 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused" Nov 02 15:47:17.340908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.340824 5433 generic.go:356] "Generic (PLEG): container finished" podID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerID="260cbda54fd6ca76b62537ce32e3533da036eb7f5a2f9ed96e06088e1302c29a" exitCode=0 Nov 02 15:47:17.341099 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.340902 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" event={"ID":"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9","Type":"ContainerDied","Data":"260cbda54fd6ca76b62537ce32e3533da036eb7f5a2f9ed96e06088e1302c29a"} Nov 02 15:47:17.342847 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.342824 5433 generic.go:356] "Generic (PLEG): container finished" podID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerID="82562aff686ef6fdeeeedb44c0c6702a68b33fcd98c21906e729670348ce7510" exitCode=0 Nov 02 15:47:17.343018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.342904 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" event={"ID":"9ee74c4b-83fe-42f7-980f-7fc179e9c569","Type":"ContainerDied","Data":"82562aff686ef6fdeeeedb44c0c6702a68b33fcd98c21906e729670348ce7510"} Nov 02 15:47:17.344689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.344655 5433 generic.go:356] "Generic (PLEG): container finished" podID="440458b0-3f93-4f8a-a9b8-8f0e321bc22c" containerID="724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa" exitCode=0 Nov 02 15:47:17.344802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.344711 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" event={"ID":"440458b0-3f93-4f8a-a9b8-8f0e321bc22c","Type":"ContainerDied","Data":"724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa"} Nov 02 15:47:17.344802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.344762 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" event={"ID":"440458b0-3f93-4f8a-a9b8-8f0e321bc22c","Type":"ContainerDied","Data":"08846d32c6a8b9e81127830d8c3878748d3120076db5c86f037e5a12aee75db0"} Nov 02 15:47:17.344921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.344801 5433 scope.go:122] "RemoveContainer" containerID="724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa" Nov 02 15:47:17.344921 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.344715 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv" Nov 02 15:47:17.355055 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.355036 5433 scope.go:122] "RemoveContainer" containerID="724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa" Nov 02 15:47:17.355355 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:17.355330 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa\": container with ID starting with 724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa not found: ID does not exist" containerID="724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa" Nov 02 15:47:17.355463 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.355377 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa"} err="failed to get container status \"724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa\": rpc error: code = NotFound desc = could not find container \"724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa\": container with ID starting with 724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa not found: ID does not exist" Nov 02 15:47:17.372794 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.372773 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:47:17.390574 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.390554 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:47:17.496295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496261 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-kube-api-access-nfcvp\" (UniqueName: \"kubernetes.io/projected/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-kube-api-access-nfcvp\") pod \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " Nov 02 15:47:17.496498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496304 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tmp-dir\") pod \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " Nov 02 15:47:17.496498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496334 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee74c4b-83fe-42f7-980f-7fc179e9c569-kube-api-access-wh7t5\" (UniqueName: \"kubernetes.io/projected/9ee74c4b-83fe-42f7-980f-7fc179e9c569-kube-api-access-wh7t5\") pod \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " Nov 02 15:47:17.496498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496361 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-model-cache\") pod \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " Nov 02 15:47:17.496498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496417 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-dshm\") pod \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " Nov 02 15:47:17.496498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496473 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-home\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-home\") pod \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " Nov 02 15:47:17.496498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496499 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-home\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-home\") pod \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " Nov 02 15:47:17.497223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496661 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tmp-dir" pod "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" (UID: "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.497223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496558 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-model-cache\") pod \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " Nov 02 15:47:17.497223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.497043 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tls-certs\") pod \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\" (UID: \"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9\") " Nov 02 15:47:17.497223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.497107 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-dshm\") pod \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " Nov 02 15:47:17.497223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.497146 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tmp-dir\") pod \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " Nov 02 15:47:17.497223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.497187 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tls-certs\") pod \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\" (UID: \"9ee74c4b-83fe-42f7-980f-7fc179e9c569\") " Nov 02 15:47:17.497763 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.497738 5433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tmp-dir\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.499402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.496939 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-home" pod "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" (UID: "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.499402 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.497226 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-model-cache" pod "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" (UID: "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.499588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.498201 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-model-cache" pod "9ee74c4b-83fe-42f7-980f-7fc179e9c569" (UID: "9ee74c4b-83fe-42f7-980f-7fc179e9c569"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.499588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.498214 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-home" pod "9ee74c4b-83fe-42f7-980f-7fc179e9c569" (UID: "9ee74c4b-83fe-42f7-980f-7fc179e9c569"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.499588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.499366 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-dshm" pod "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" (UID: "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.499588 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.499492 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-kube-api-access-nfcvp" pod "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" (UID: "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9"). InnerVolumeSpecName "kube-api-access-nfcvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:17.499811 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.499647 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tmp-dir" pod "9ee74c4b-83fe-42f7-980f-7fc179e9c569" (UID: "9ee74c4b-83fe-42f7-980f-7fc179e9c569"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.499871 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.499845 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee74c4b-83fe-42f7-980f-7fc179e9c569-kube-api-access-wh7t5" pod "9ee74c4b-83fe-42f7-980f-7fc179e9c569" (UID: "9ee74c4b-83fe-42f7-980f-7fc179e9c569"). InnerVolumeSpecName "kube-api-access-wh7t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:17.500329 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.500305 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-dshm" pod "9ee74c4b-83fe-42f7-980f-7fc179e9c569" (UID: "9ee74c4b-83fe-42f7-980f-7fc179e9c569"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.500329 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.500320 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv"] Nov 02 15:47:17.500493 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.500428 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tls-certs" pod "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" (UID: "fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:17.500493 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.500433 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tls-certs" pod "9ee74c4b-83fe-42f7-980f-7fc179e9c569" (UID: "9ee74c4b-83fe-42f7-980f-7fc179e9c569"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:17.546238 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.546205 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe8ef1-5bc47d785c-9s7tv"] Nov 02 15:47:17.598817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598733 5433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-home\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.598817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598774 5433 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-home\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.598817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598789 5433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-model-cache\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.598817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598804 5433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-tls-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.598817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598820 5433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-dshm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.599201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598836 5433 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tmp-dir\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.599201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598849 5433 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee74c4b-83fe-42f7-980f-7fc179e9c569-tls-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.599201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598864 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfcvp\" (UniqueName: \"kubernetes.io/projected/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-kube-api-access-nfcvp\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.599201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598881 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh7t5\" (UniqueName: \"kubernetes.io/projected/9ee74c4b-83fe-42f7-980f-7fc179e9c569-kube-api-access-wh7t5\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.599201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598896 5433 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-model-cache\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.599201 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:17.598911 5433 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9-dshm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:18.231942 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.231908 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:18.274172 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.274140 5433 scope.go:122] "RemoveContainer" containerID="82562aff686ef6fdeeeedb44c0c6702a68b33fcd98c21906e729670348ce7510" Nov 02 15:47:18.282803 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.282781 5433 scope.go:122] "RemoveContainer" containerID="260cbda54fd6ca76b62537ce32e3533da036eb7f5a2f9ed96e06088e1302c29a" Nov 02 15:47:18.349077 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.349049 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" Nov 02 15:47:18.349178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.349067 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" event={"ID":"fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9","Type":"ContainerDied","Data":"e0775e6535e76a4a696af6374cbdf877c5174ea03e3fbc52a638c4b7263cf7a3"} Nov 02 15:47:18.349178 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.349106 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7" event={"ID":"9ee74c4b-83fe-42f7-980f-7fc179e9c569","Type":"ContainerDied","Data":"e41e58f5a9504b485af9d732e6716bea4605c618b381b08dee6bf175a32edb51"} Nov 02 15:47:18.349294 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.349188 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl" Nov 02 15:47:18.455863 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.455817 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:18.456770 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.456733 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440458b0-3f93-4f8a-a9b8-8f0e321bc22c" path="/var/lib/kubelet/pods/440458b0-3f93-4f8a-a9b8-8f0e321bc22c/volumes" Nov 02 15:47:18.564208 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.564116 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7"] Nov 02 15:47:18.565689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.565668 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/premium-w6-e98976-kserve-5b7b8b9bf6-r4jb7"] Nov 02 15:47:18.595853 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.595820 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:18.732694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.732660 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl"] Nov 02 15:47:18.736320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:18.736296 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-w6-e98976-kserve-866b4b8767-t77cl"] Nov 02 15:47:18.822229 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:18.822156 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:19.186604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:19.186567 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:19.325677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:19.325637 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:19.652200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:19.652169 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:20.455911 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:20.455877 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" path="/var/lib/kubelet/pods/9ee74c4b-83fe-42f7-980f-7fc179e9c569/volumes" Nov 02 15:47:20.458231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:20.456357 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" path="/var/lib/kubelet/pods/fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9/volumes" Nov 02 15:47:23.194787 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:23.194756 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:23.608692 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:23.608606 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:23.624007 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:23.623949 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" podUID="373b0af7-e073-474c-b465-8c0b22b359af" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:24.629228 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:24.629190 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" podUID="500c39b4-df23-4fe6-abeb-46b7b9172de2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:24.918679 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:24.918586 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:25.173417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:25.173312 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:25.316177 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:25.316140 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:25.847367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:25.847311 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:26.044936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:26.044885 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:26.190164 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:26.190131 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:26.425516 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:26.425475 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:26.430012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:26.429970 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc"] Nov 02 15:47:26.430248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:26.430210 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" containerID="cri-o://80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e" gracePeriod=30 Nov 02 15:47:26.433783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:26.433758 5433 patch_prober.go:29] interesting pod/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:47:26.433904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:26.433801 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:47:26.889485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:26.889454 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:28.879011 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:28.878967 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:30.678578 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:30.678544 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:31.359412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.359379 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd"] Nov 02 15:47:31.605756 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.605725 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:31.710061 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.710032 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:47:31.745565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.745539 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:31.768390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768360 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-credential-socket\") pod \"c4120f89-4395-46f6-a62c-3fb79150b20e\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " Nov 02 15:47:31.768518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768425 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-data\") pod \"c4120f89-4395-46f6-a62c-3fb79150b20e\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " Nov 02 15:47:31.768518 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768491 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4120f89-4395-46f6-a62c-3fb79150b20e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4120f89-4395-46f6-a62c-3fb79150b20e-istiod-ca-cert\") pod \"c4120f89-4395-46f6-a62c-3fb79150b20e\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " Nov 02 15:47:31.768618 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768522 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-envoy\") pod \"c4120f89-4395-46f6-a62c-3fb79150b20e\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " Nov 02 15:47:31.768618 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768560 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-kube-api-access-wfnsm\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-kube-api-access-wfnsm\") pod \"c4120f89-4395-46f6-a62c-3fb79150b20e\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " Nov 02 15:47:31.768711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768617 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-certs\") pod \"c4120f89-4395-46f6-a62c-3fb79150b20e\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " Nov 02 15:47:31.768711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768654 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-token\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-token\") pod \"c4120f89-4395-46f6-a62c-3fb79150b20e\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " Nov 02 15:47:31.768809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768723 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-data" pod "c4120f89-4395-46f6-a62c-3fb79150b20e" (UID: "c4120f89-4395-46f6-a62c-3fb79150b20e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:31.768809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768739 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-socket\") pod \"c4120f89-4395-46f6-a62c-3fb79150b20e\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " Nov 02 15:47:31.768809 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768782 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-podinfo\") pod \"c4120f89-4395-46f6-a62c-3fb79150b20e\" (UID: \"c4120f89-4395-46f6-a62c-3fb79150b20e\") " Nov 02 15:47:31.769011 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.768924 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-certs" pod "c4120f89-4395-46f6-a62c-3fb79150b20e" (UID: "c4120f89-4395-46f6-a62c-3fb79150b20e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:31.769088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.769025 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4120f89-4395-46f6-a62c-3fb79150b20e-istiod-ca-cert" pod "c4120f89-4395-46f6-a62c-3fb79150b20e" (UID: "c4120f89-4395-46f6-a62c-3fb79150b20e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:31.769088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.769046 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-socket" pod "c4120f89-4395-46f6-a62c-3fb79150b20e" (UID: "c4120f89-4395-46f6-a62c-3fb79150b20e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:31.769088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.769056 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-credential-socket" pod "c4120f89-4395-46f6-a62c-3fb79150b20e" (UID: "c4120f89-4395-46f6-a62c-3fb79150b20e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:31.769223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.769191 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:31.769223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.769203 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:31.769223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.769212 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4120f89-4395-46f6-a62c-3fb79150b20e-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:31.769223 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.769222 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:31.769443 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.769234 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:31.771128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.771098 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-token" pod "c4120f89-4395-46f6-a62c-3fb79150b20e" (UID: "c4120f89-4395-46f6-a62c-3fb79150b20e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:31.771128 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.771107 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-podinfo" pod "c4120f89-4395-46f6-a62c-3fb79150b20e" (UID: "c4120f89-4395-46f6-a62c-3fb79150b20e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:47:31.771605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.771247 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-envoy" pod "c4120f89-4395-46f6-a62c-3fb79150b20e" (UID: "c4120f89-4395-46f6-a62c-3fb79150b20e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:31.771605 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.771490 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-kube-api-access-wfnsm" pod "c4120f89-4395-46f6-a62c-3fb79150b20e" (UID: "c4120f89-4395-46f6-a62c-3fb79150b20e"). InnerVolumeSpecName "kube-api-access-wfnsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:31.870689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.870655 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:31.870689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.870686 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:31.870689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.870696 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfnsm\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-kube-api-access-wfnsm\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:31.870938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:31.870707 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4120f89-4395-46f6-a62c-3fb79150b20e-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:32.047258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.047170 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:32.424610 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.424566 5433 generic.go:356] "Generic (PLEG): container finished" podID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerID="80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e" exitCode=0 Nov 02 15:47:32.424791 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.424622 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" Nov 02 15:47:32.424791 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.424650 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" event={"ID":"c4120f89-4395-46f6-a62c-3fb79150b20e","Type":"ContainerDied","Data":"80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e"} Nov 02 15:47:32.424791 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.424707 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc" event={"ID":"c4120f89-4395-46f6-a62c-3fb79150b20e","Type":"ContainerDied","Data":"71f9eb0229c2ca8925dba3ed1155d5c6a7c3f6c48d85a3b04d66af076b10311d"} Nov 02 15:47:32.424791 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.424732 5433 scope.go:122] "RemoveContainer" containerID="80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e" Nov 02 15:47:32.434632 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.434607 5433 scope.go:122] "RemoveContainer" containerID="0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6" Nov 02 15:47:32.443410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.443384 5433 scope.go:122] "RemoveContainer" containerID="80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e" Nov 02 15:47:32.443678 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:32.443658 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e\": container with ID starting with 80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e not found: ID does not exist" containerID="80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e" Nov 02 15:47:32.443759 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.443689 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e"} err="failed to get container status \"80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e\": rpc error: code = NotFound desc = could not find container \"80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e\": container with ID starting with 80b576c828d2a4294e80b3f5224f06d856a33b2a447bdd408743a95554d16d9e not found: ID does not exist" Nov 02 15:47:32.443759 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.443716 5433 scope.go:122] "RemoveContainer" containerID="0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6" Nov 02 15:47:32.444065 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:32.443950 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6\": container with ID starting with 0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6 not found: ID does not exist" containerID="0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6" Nov 02 15:47:32.444124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.444058 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6"} err="failed to get container status \"0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6\": rpc error: code = NotFound desc = could not find container \"0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6\": container with ID starting with 0be6ad198b12fdbf2d256889620652ddd1130de370e9122704fc6154cfa7b0d6 not found: ID does not exist" Nov 02 15:47:32.580039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.580004 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc"] Nov 02 15:47:32.584085 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.584059 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fe8ef1-openshift-default-54cd7cfc7c-t9nsc"] Nov 02 15:47:32.787991 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.787899 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:32.929231 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:32.929198 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:33.122324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:33.122244 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:33.609621 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:33.609585 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:33.624612 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:33.624584 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" podUID="373b0af7-e073-474c-b465-8c0b22b359af" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:34.353159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.353135 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:47:34.394808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.393612 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500c39b4-df23-4fe6-abeb-46b7b9172de2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500c39b4-df23-4fe6-abeb-46b7b9172de2-plugins-config-volume\") pod \"500c39b4-df23-4fe6-abeb-46b7b9172de2\" (UID: \"500c39b4-df23-4fe6-abeb-46b7b9172de2\") " Nov 02 15:47:34.394808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.393668 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500c39b4-df23-4fe6-abeb-46b7b9172de2-kube-api-access-72h5x\" (UniqueName: \"kubernetes.io/projected/500c39b4-df23-4fe6-abeb-46b7b9172de2-kube-api-access-72h5x\") pod \"500c39b4-df23-4fe6-abeb-46b7b9172de2\" (UID: \"500c39b4-df23-4fe6-abeb-46b7b9172de2\") " Nov 02 15:47:34.394808 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.395266 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500c39b4-df23-4fe6-abeb-46b7b9172de2-plugins-config-volume" pod "500c39b4-df23-4fe6-abeb-46b7b9172de2" (UID: "500c39b4-df23-4fe6-abeb-46b7b9172de2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:34.415382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.415324 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500c39b4-df23-4fe6-abeb-46b7b9172de2-kube-api-access-72h5x" pod "500c39b4-df23-4fe6-abeb-46b7b9172de2" (UID: "500c39b4-df23-4fe6-abeb-46b7b9172de2"). InnerVolumeSpecName "kube-api-access-72h5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:34.436210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.436181 5433 generic.go:356] "Generic (PLEG): container finished" podID="500c39b4-df23-4fe6-abeb-46b7b9172de2" containerID="ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444" exitCode=0 Nov 02 15:47:34.436366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.436243 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" Nov 02 15:47:34.436366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.436265 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" event={"ID":"500c39b4-df23-4fe6-abeb-46b7b9172de2","Type":"ContainerDied","Data":"ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444"} Nov 02 15:47:34.436366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.436332 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r" event={"ID":"500c39b4-df23-4fe6-abeb-46b7b9172de2","Type":"ContainerDied","Data":"f519aba9e9fa059079db9bb84fb80a01a03329700670e72dd2a70de96ef80637"} Nov 02 15:47:34.436366 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.436361 5433 scope.go:122] "RemoveContainer" containerID="ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444" Nov 02 15:47:34.438429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.438325 5433 generic.go:356] "Generic (PLEG): container finished" podID="373b0af7-e073-474c-b465-8c0b22b359af" containerID="4d793d05a38e8a1efa198326dda70bcecc3d3f47e35ea023f94c23e9f40353d6" exitCode=0 Nov 02 15:47:34.438429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.438376 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" event={"ID":"373b0af7-e073-474c-b465-8c0b22b359af","Type":"ContainerDied","Data":"4d793d05a38e8a1efa198326dda70bcecc3d3f47e35ea023f94c23e9f40353d6"} Nov 02 15:47:34.446272 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.446257 5433 scope.go:122] "RemoveContainer" containerID="ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444" Nov 02 15:47:34.446632 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:34.446607 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444\": container with ID starting with ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444 not found: ID does not exist" containerID="ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444" Nov 02 15:47:34.446790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.446764 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444"} err="failed to get container status \"ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444\": rpc error: code = NotFound desc = could not find container \"ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444\": container with ID starting with ecd1faeb6551fca080e77cfedd7c06e2de1df61b599a7634593a4d745f3cd444 not found: ID does not exist" Nov 02 15:47:34.455698 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.455676 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" path="/var/lib/kubelet/pods/c4120f89-4395-46f6-a62c-3fb79150b20e/volumes" Nov 02 15:47:34.467832 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.467792 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:47:34.494766 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.494739 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373b0af7-e073-474c-b465-8c0b22b359af-kube-api-access-798z5\" (UniqueName: \"kubernetes.io/projected/373b0af7-e073-474c-b465-8c0b22b359af-kube-api-access-798z5\") pod \"373b0af7-e073-474c-b465-8c0b22b359af\" (UID: \"373b0af7-e073-474c-b465-8c0b22b359af\") " Nov 02 15:47:34.494898 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.494821 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373b0af7-e073-474c-b465-8c0b22b359af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373b0af7-e073-474c-b465-8c0b22b359af-plugins-config-volume\") pod \"373b0af7-e073-474c-b465-8c0b22b359af\" (UID: \"373b0af7-e073-474c-b465-8c0b22b359af\") " Nov 02 15:47:34.495152 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.495129 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500c39b4-df23-4fe6-abeb-46b7b9172de2-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:34.495245 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.495156 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72h5x\" (UniqueName: \"kubernetes.io/projected/500c39b4-df23-4fe6-abeb-46b7b9172de2-kube-api-access-72h5x\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:34.495245 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.495197 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373b0af7-e073-474c-b465-8c0b22b359af-plugins-config-volume" pod "373b0af7-e073-474c-b465-8c0b22b359af" (UID: "373b0af7-e073-474c-b465-8c0b22b359af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:34.496864 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.496845 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373b0af7-e073-474c-b465-8c0b22b359af-kube-api-access-798z5" pod "373b0af7-e073-474c-b465-8c0b22b359af" (UID: "373b0af7-e073-474c-b465-8c0b22b359af"). InnerVolumeSpecName "kube-api-access-798z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:34.569096 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.569065 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r"] Nov 02 15:47:34.573048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.573019 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-e98976-6799864d59-b6g2r"] Nov 02 15:47:34.595686 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.595663 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-798z5\" (UniqueName: \"kubernetes.io/projected/373b0af7-e073-474c-b465-8c0b22b359af-kube-api-access-798z5\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:34.595686 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:34.595684 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373b0af7-e073-474c-b465-8c0b22b359af-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:35.445522 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:35.445492 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" Nov 02 15:47:35.445970 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:35.445535 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk" event={"ID":"373b0af7-e073-474c-b465-8c0b22b359af","Type":"ContainerDied","Data":"b257b7e2dfbd1f5680102228b2557aec8f7e61e77fd7d180396f191615097b20"} Nov 02 15:47:35.445970 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:35.445595 5433 scope.go:122] "RemoveContainer" containerID="4d793d05a38e8a1efa198326dda70bcecc3d3f47e35ea023f94c23e9f40353d6" Nov 02 15:47:35.600048 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:35.600013 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk"] Nov 02 15:47:35.603970 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:35.603941 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-e98976-86c8d48487-2tsbk"] Nov 02 15:47:35.829457 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:35.829373 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:35.829457 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:35.829408 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:36.090312 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:36.090230 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:36.360255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:36.360167 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" podUID="2b2267b7-0f71-412b-b161-1d2425f884fa" containerName="payload-pre-processing" containerID="cri-o://331cd79b6f42172014e192f35695112ca177a23c7d9a79bdb6ba526dd5aaf545" gracePeriod=25 Nov 02 15:47:36.455911 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:36.455879 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373b0af7-e073-474c-b465-8c0b22b359af" path="/var/lib/kubelet/pods/373b0af7-e073-474c-b465-8c0b22b359af/volumes" Nov 02 15:47:36.456293 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:36.456279 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500c39b4-df23-4fe6-abeb-46b7b9172de2" path="/var/lib/kubelet/pods/500c39b4-df23-4fe6-abeb-46b7b9172de2/volumes" Nov 02 15:47:38.587456 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:38.587421 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:38.613944 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:38.613908 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:38.729966 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:38.729913 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:38.896917 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:38.896874 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:39.734491 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:39.734452 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:39.878130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:39.878092 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:40.226344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:40.226308 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:41.012068 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:41.012031 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:47:42.020032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:42.019996 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:44.307255 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:44.307219 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" podUID="2b2267b7-0f71-412b-b161-1d2425f884fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:46.061085 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:46.061044 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:46.202540 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:46.202505 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:46.350888 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:46.350806 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:47.739579 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:47.739538 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:48.951237 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:48.951203 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:50.375446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:50.375413 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:50.529604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:50.529573 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:51.518215 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:51.518150 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:52.284326 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:52.284291 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:53.600272 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:53.600227 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:53.920429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:53.920395 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:54.161175 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:54.161126 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:54.304711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:54.304621 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:54.305661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:54.305638 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" podUID="2b2267b7-0f71-412b-b161-1d2425f884fa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:54.880320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:54.880281 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:56.559042 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:56.559008 5433 generic.go:356] "Generic (PLEG): container finished" podID="2b2267b7-0f71-412b-b161-1d2425f884fa" containerID="331cd79b6f42172014e192f35695112ca177a23c7d9a79bdb6ba526dd5aaf545" exitCode=0 Nov 02 15:47:56.559483 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:56.559162 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" event={"ID":"2b2267b7-0f71-412b-b161-1d2425f884fa","Type":"ContainerDied","Data":"331cd79b6f42172014e192f35695112ca177a23c7d9a79bdb6ba526dd5aaf545"} Nov 02 15:47:56.620067 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:56.620007 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:47:56.734230 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:56.734201 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2267b7-0f71-412b-b161-1d2425f884fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2267b7-0f71-412b-b161-1d2425f884fa-plugins-config-volume\") pod \"2b2267b7-0f71-412b-b161-1d2425f884fa\" (UID: \"2b2267b7-0f71-412b-b161-1d2425f884fa\") " Nov 02 15:47:56.734412 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:56.734358 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2267b7-0f71-412b-b161-1d2425f884fa-kube-api-access-2mh7z\" (UniqueName: \"kubernetes.io/projected/2b2267b7-0f71-412b-b161-1d2425f884fa-kube-api-access-2mh7z\") pod \"2b2267b7-0f71-412b-b161-1d2425f884fa\" (UID: \"2b2267b7-0f71-412b-b161-1d2425f884fa\") " Nov 02 15:47:56.734660 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:56.734634 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2267b7-0f71-412b-b161-1d2425f884fa-plugins-config-volume" pod "2b2267b7-0f71-412b-b161-1d2425f884fa" (UID: "2b2267b7-0f71-412b-b161-1d2425f884fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:56.736829 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:56.736794 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2267b7-0f71-412b-b161-1d2425f884fa-kube-api-access-2mh7z" pod "2b2267b7-0f71-412b-b161-1d2425f884fa" (UID: "2b2267b7-0f71-412b-b161-1d2425f884fa"). InnerVolumeSpecName "kube-api-access-2mh7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:56.835641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:56.835607 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mh7z\" (UniqueName: \"kubernetes.io/projected/2b2267b7-0f71-412b-b161-1d2425f884fa-kube-api-access-2mh7z\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:56.835641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:56.835635 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2267b7-0f71-412b-b161-1d2425f884fa-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:57.565446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:57.565405 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" Nov 02 15:47:57.565446 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:57.565419 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd" event={"ID":"2b2267b7-0f71-412b-b161-1d2425f884fa","Type":"ContainerDied","Data":"1e71e40ac534021be48d489043b1d5e9b8970b0146ba58a0714bc527a22fd559"} Nov 02 15:47:57.566032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:57.565503 5433 scope.go:122] "RemoveContainer" containerID="331cd79b6f42172014e192f35695112ca177a23c7d9a79bdb6ba526dd5aaf545" Nov 02 15:47:57.720614 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:57.720578 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd"] Nov 02 15:47:57.767416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:57.767379 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f6b245-644c56b5f-j7wxd"] Nov 02 15:47:58.005777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:58.005740 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:47:58.456895 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:58.456859 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2267b7-0f71-412b-b161-1d2425f884fa" path="/var/lib/kubelet/pods/2b2267b7-0f71-412b-b161-1d2425f884fa/volumes" Nov 02 15:47:59.004157 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:47:59.004122 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:59.707504 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:47:59.707461 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:01.647531 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:01.647493 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:01.826708 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:01.826671 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:02.057936 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:02.057851 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b"] Nov 02 15:48:02.058119 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:02.058095 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" containerID="cri-o://20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63" gracePeriod=30 Nov 02 15:48:02.065667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:02.065616 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:04.775715 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:04.775662 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:06.536354 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:06.536314 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:06.779862 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:06.779827 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:06.987129 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:06.987097 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:07.316744 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.316720 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:48:07.337429 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337399 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-kube-api-access-j577t\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-kube-api-access-j577t\") pod \"c978cb20-4d48-48db-a682-f8d9916a460e\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " Nov 02 15:48:07.337549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337435 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-credential-socket\") pod \"c978cb20-4d48-48db-a682-f8d9916a460e\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " Nov 02 15:48:07.337549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337472 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c978cb20-4d48-48db-a682-f8d9916a460e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c978cb20-4d48-48db-a682-f8d9916a460e-istio-podinfo\") pod \"c978cb20-4d48-48db-a682-f8d9916a460e\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " Nov 02 15:48:07.337549 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337492 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-data\") pod \"c978cb20-4d48-48db-a682-f8d9916a460e\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " Nov 02 15:48:07.337685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337559 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-istio-token\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-istio-token\") pod \"c978cb20-4d48-48db-a682-f8d9916a460e\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " Nov 02 15:48:07.337685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337590 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-certs\") pod \"c978cb20-4d48-48db-a682-f8d9916a460e\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " Nov 02 15:48:07.337685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337617 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c978cb20-4d48-48db-a682-f8d9916a460e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c978cb20-4d48-48db-a682-f8d9916a460e-istiod-ca-cert\") pod \"c978cb20-4d48-48db-a682-f8d9916a460e\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " Nov 02 15:48:07.337685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337651 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-socket\") pod \"c978cb20-4d48-48db-a682-f8d9916a460e\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " Nov 02 15:48:07.337685 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337679 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-envoy\") pod \"c978cb20-4d48-48db-a682-f8d9916a460e\" (UID: \"c978cb20-4d48-48db-a682-f8d9916a460e\") " Nov 02 15:48:07.337960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337795 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-data" pod "c978cb20-4d48-48db-a682-f8d9916a460e" (UID: "c978cb20-4d48-48db-a682-f8d9916a460e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:07.337960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337812 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-credential-socket" pod "c978cb20-4d48-48db-a682-f8d9916a460e" (UID: "c978cb20-4d48-48db-a682-f8d9916a460e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:07.337960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.337891 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-certs" pod "c978cb20-4d48-48db-a682-f8d9916a460e" (UID: "c978cb20-4d48-48db-a682-f8d9916a460e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:07.338130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.338069 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c978cb20-4d48-48db-a682-f8d9916a460e-istiod-ca-cert" pod "c978cb20-4d48-48db-a682-f8d9916a460e" (UID: "c978cb20-4d48-48db-a682-f8d9916a460e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:07.338210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.338144 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.338267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.338214 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.338267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.338230 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.338267 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.338236 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-socket" pod "c978cb20-4d48-48db-a682-f8d9916a460e" (UID: "c978cb20-4d48-48db-a682-f8d9916a460e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:07.340005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.339951 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-kube-api-access-j577t" pod "c978cb20-4d48-48db-a682-f8d9916a460e" (UID: "c978cb20-4d48-48db-a682-f8d9916a460e"). InnerVolumeSpecName "kube-api-access-j577t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:07.340186 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.340132 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-envoy" pod "c978cb20-4d48-48db-a682-f8d9916a460e" (UID: "c978cb20-4d48-48db-a682-f8d9916a460e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:07.340186 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.340149 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c978cb20-4d48-48db-a682-f8d9916a460e-istio-podinfo" pod "c978cb20-4d48-48db-a682-f8d9916a460e" (UID: "c978cb20-4d48-48db-a682-f8d9916a460e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:48:07.340317 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.340225 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-istio-token" pod "c978cb20-4d48-48db-a682-f8d9916a460e" (UID: "c978cb20-4d48-48db-a682-f8d9916a460e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:07.439208 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.439174 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.439208 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.439201 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c978cb20-4d48-48db-a682-f8d9916a460e-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.439208 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.439212 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.439466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.439224 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c978cb20-4d48-48db-a682-f8d9916a460e-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.439466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.439234 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j577t\" (UniqueName: \"kubernetes.io/projected/c978cb20-4d48-48db-a682-f8d9916a460e-kube-api-access-j577t\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.439466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.439244 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c978cb20-4d48-48db-a682-f8d9916a460e-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.563960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.563866 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:07.619000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.618944 5433 generic.go:356] "Generic (PLEG): container finished" podID="c978cb20-4d48-48db-a682-f8d9916a460e" containerID="20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63" exitCode=0 Nov 02 15:48:07.619171 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.619027 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" Nov 02 15:48:07.619171 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.619025 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" event={"ID":"c978cb20-4d48-48db-a682-f8d9916a460e","Type":"ContainerDied","Data":"20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63"} Nov 02 15:48:07.619171 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.619083 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b" event={"ID":"c978cb20-4d48-48db-a682-f8d9916a460e","Type":"ContainerDied","Data":"7a2dc96bf56d16916195b76807aff81e4726aead5661af8b24fd1b83c5f282d3"} Nov 02 15:48:07.619171 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.619105 5433 scope.go:122] "RemoveContainer" containerID="20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63" Nov 02 15:48:07.628952 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.628928 5433 scope.go:122] "RemoveContainer" containerID="b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249" Nov 02 15:48:07.637881 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.637860 5433 scope.go:122] "RemoveContainer" containerID="20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63" Nov 02 15:48:07.638200 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:07.638179 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63\": container with ID starting with 20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63 not found: ID does not exist" containerID="20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63" Nov 02 15:48:07.638252 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.638212 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63"} err="failed to get container status \"20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63\": rpc error: code = NotFound desc = could not find container \"20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63\": container with ID starting with 20b6dc6d611283d13191cc975aff3498e8799b1dd1a7adf585ae1a07a0998c63 not found: ID does not exist" Nov 02 15:48:07.638252 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.638236 5433 scope.go:122] "RemoveContainer" containerID="b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249" Nov 02 15:48:07.638551 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:07.638500 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249\": container with ID starting with b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249 not found: ID does not exist" containerID="b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249" Nov 02 15:48:07.638551 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.638526 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249"} err="failed to get container status \"b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249\": rpc error: code = NotFound desc = could not find container \"b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249\": container with ID starting with b1a168207787e53504c176c05f420f568f9622961fe1a593f18221f2b66c3249 not found: ID does not exist" Nov 02 15:48:07.774879 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.774839 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b"] Nov 02 15:48:07.781151 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:07.781118 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f6b245-openshift-default-797c5cb9f9-bx58b"] Nov 02 15:48:08.455908 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:08.455860 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" path="/var/lib/kubelet/pods/c978cb20-4d48-48db-a682-f8d9916a460e/volumes" Nov 02 15:48:08.652766 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:08.652733 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:48:09.016391 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:09.016347 5433 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440458b0_3f93_4f8a_a9b8_8f0e321bc22c.slice/crio-724306e13d8d96cf974e23b8f2280c2502e8475c368bad0f2fdc97a9ca40d5fa.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:48:11.010930 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:11.010893 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:48:11.653417 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:11.653382 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:13.321570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:13.321534 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:13.603189 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:13.603105 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:13.741090 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:13.741056 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:13.890899 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:13.890868 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:14.147383 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:14.146510 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:14.292960 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:14.292917 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:15.394276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:15.394235 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:15.583353 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:15.583301 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:18.351539 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.351515 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q"] Nov 02 15:48:18.351904 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.351699 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" podUID="b2aa573f-e7f4-4b63-9615-98072075a183" containerName="maas-api" containerID="cri-o://330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28" gracePeriod=30 Nov 02 15:48:18.450864 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.450834 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:18.600890 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.600829 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:48:18.651423 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.651390 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2aa573f-e7f4-4b63-9615-98072075a183-kube-api-access-phpmd\" (UniqueName: \"kubernetes.io/projected/b2aa573f-e7f4-4b63-9615-98072075a183-kube-api-access-phpmd\") pod \"b2aa573f-e7f4-4b63-9615-98072075a183\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " Nov 02 15:48:18.651599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.651457 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls\") pod \"b2aa573f-e7f4-4b63-9615-98072075a183\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " Nov 02 15:48:18.651599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.651510 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-metrics-certs\") pod \"b2aa573f-e7f4-4b63-9615-98072075a183\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " Nov 02 15:48:18.651599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.651543 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2aa573f-e7f4-4b63-9615-98072075a183-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aa573f-e7f4-4b63-9615-98072075a183-openshift-service-ca\") pod \"b2aa573f-e7f4-4b63-9615-98072075a183\" (UID: \"b2aa573f-e7f4-4b63-9615-98072075a183\") " Nov 02 15:48:18.651992 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.651944 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2aa573f-e7f4-4b63-9615-98072075a183-openshift-service-ca" pod "b2aa573f-e7f4-4b63-9615-98072075a183" (UID: "b2aa573f-e7f4-4b63-9615-98072075a183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:18.653586 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.653558 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-metrics-certs" pod "b2aa573f-e7f4-4b63-9615-98072075a183" (UID: "b2aa573f-e7f4-4b63-9615-98072075a183"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:18.653680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.653620 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls" pod "b2aa573f-e7f4-4b63-9615-98072075a183" (UID: "b2aa573f-e7f4-4b63-9615-98072075a183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:18.653680 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.653634 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2aa573f-e7f4-4b63-9615-98072075a183-kube-api-access-phpmd" pod "b2aa573f-e7f4-4b63-9615-98072075a183" (UID: "b2aa573f-e7f4-4b63-9615-98072075a183"). InnerVolumeSpecName "kube-api-access-phpmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:18.675263 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.675239 5433 generic.go:356] "Generic (PLEG): container finished" podID="b2aa573f-e7f4-4b63-9615-98072075a183" containerID="330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28" exitCode=0 Nov 02 15:48:18.675364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.675275 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" event={"ID":"b2aa573f-e7f4-4b63-9615-98072075a183","Type":"ContainerDied","Data":"330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28"} Nov 02 15:48:18.675364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.675283 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" Nov 02 15:48:18.675364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.675313 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q" event={"ID":"b2aa573f-e7f4-4b63-9615-98072075a183","Type":"ContainerDied","Data":"7120006c50dcdde5397e728107ff612663196564aff05137cc18543bda18a32b"} Nov 02 15:48:18.675364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.675334 5433 scope.go:122] "RemoveContainer" containerID="330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28" Nov 02 15:48:18.684289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.684273 5433 scope.go:122] "RemoveContainer" containerID="330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28" Nov 02 15:48:18.684578 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:18.684549 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28\": container with ID starting with 330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28 not found: ID does not exist" containerID="330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28" Nov 02 15:48:18.684664 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.684587 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28"} err="failed to get container status \"330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28\": rpc error: code = NotFound desc = could not find container \"330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28\": container with ID starting with 330adabbf981aef6681126b2a7c3bd63eae6acc0ff0c05635b3347a97a427d28 not found: ID does not exist" Nov 02 15:48:18.752439 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.752414 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phpmd\" (UniqueName: \"kubernetes.io/projected/b2aa573f-e7f4-4b63-9615-98072075a183-kube-api-access-phpmd\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:18.752439 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.752438 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:18.752613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.752449 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2aa573f-e7f4-4b63-9615-98072075a183-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:18.752613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.752458 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aa573f-e7f4-4b63-9615-98072075a183-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:18.845209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.845180 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q"] Nov 02 15:48:18.850458 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:18.850416 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-7ad552-75cc9cb9c4-vr75q"] Nov 02 15:48:20.048796 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:20.048758 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:20.285878 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:20.285824 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:20.428303 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:20.428264 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:20.455724 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:20.455692 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2aa573f-e7f4-4b63-9615-98072075a183" path="/var/lib/kubelet/pods/b2aa573f-e7f4-4b63-9615-98072075a183/volumes" Nov 02 15:48:20.575490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:20.575392 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:20.730064 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:20.729941 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:20.873852 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:20.873822 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:21.016564 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:21.016479 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:22.879774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:22.879724 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:23.763018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:23.762942 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:23.902344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:23.902310 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:24.759355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:24.759315 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8"] Nov 02 15:48:24.759548 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:24.759500 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" podUID="9dc2d51c-760f-447d-8a31-675d6dae2424" containerName="maas-api" containerID="cri-o://b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd" gracePeriod=30 Nov 02 15:48:24.864727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:24.864687 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:25.003390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.003364 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:48:25.116114 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.116033 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc2d51c-760f-447d-8a31-675d6dae2424-kube-api-access-wjdt6\" (UniqueName: \"kubernetes.io/projected/9dc2d51c-760f-447d-8a31-675d6dae2424-kube-api-access-wjdt6\") pod \"9dc2d51c-760f-447d-8a31-675d6dae2424\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " Nov 02 15:48:25.116290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.116155 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc2d51c-760f-447d-8a31-675d6dae2424-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2d51c-760f-447d-8a31-675d6dae2424-openshift-service-ca\") pod \"9dc2d51c-760f-447d-8a31-675d6dae2424\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " Nov 02 15:48:25.116290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.116190 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-metrics-certs\") pod \"9dc2d51c-760f-447d-8a31-675d6dae2424\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " Nov 02 15:48:25.116290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.116230 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls\") pod \"9dc2d51c-760f-447d-8a31-675d6dae2424\" (UID: \"9dc2d51c-760f-447d-8a31-675d6dae2424\") " Nov 02 15:48:25.116660 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.116613 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc2d51c-760f-447d-8a31-675d6dae2424-openshift-service-ca" pod "9dc2d51c-760f-447d-8a31-675d6dae2424" (UID: "9dc2d51c-760f-447d-8a31-675d6dae2424"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:25.119504 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.119465 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-metrics-certs" pod "9dc2d51c-760f-447d-8a31-675d6dae2424" (UID: "9dc2d51c-760f-447d-8a31-675d6dae2424"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:25.119750 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.119717 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc2d51c-760f-447d-8a31-675d6dae2424-kube-api-access-wjdt6" pod "9dc2d51c-760f-447d-8a31-675d6dae2424" (UID: "9dc2d51c-760f-447d-8a31-675d6dae2424"). InnerVolumeSpecName "kube-api-access-wjdt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:25.120568 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.120542 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls" pod "9dc2d51c-760f-447d-8a31-675d6dae2424" (UID: "9dc2d51c-760f-447d-8a31-675d6dae2424"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:25.217987 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.217942 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjdt6\" (UniqueName: \"kubernetes.io/projected/9dc2d51c-760f-447d-8a31-675d6dae2424-kube-api-access-wjdt6\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:25.217987 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.217971 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2d51c-760f-447d-8a31-675d6dae2424-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:25.217987 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.217993 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:25.218204 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.218003 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc2d51c-760f-447d-8a31-675d6dae2424-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:25.711203 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.711171 5433 generic.go:356] "Generic (PLEG): container finished" podID="9dc2d51c-760f-447d-8a31-675d6dae2424" containerID="b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd" exitCode=0 Nov 02 15:48:25.711390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.711220 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" Nov 02 15:48:25.711390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.711258 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" event={"ID":"9dc2d51c-760f-447d-8a31-675d6dae2424","Type":"ContainerDied","Data":"b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd"} Nov 02 15:48:25.711390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.711319 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8" event={"ID":"9dc2d51c-760f-447d-8a31-675d6dae2424","Type":"ContainerDied","Data":"75034a385d63b40bd1c28628397d84c68f476bf76a7b935224fa7bf074fbdf25"} Nov 02 15:48:25.711390 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.711355 5433 scope.go:122] "RemoveContainer" containerID="b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd" Nov 02 15:48:25.729076 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.729038 5433 scope.go:122] "RemoveContainer" containerID="b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd" Nov 02 15:48:25.729415 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:25.729389 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd\": container with ID starting with b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd not found: ID does not exist" containerID="b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd" Nov 02 15:48:25.729498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.729430 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd"} err="failed to get container status \"b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd\": rpc error: code = NotFound desc = could not find container \"b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd\": container with ID starting with b9aff3f2160fb876102e08a5864e3192ad3c1e7bc130992da6f734dd335485dd not found: ID does not exist" Nov 02 15:48:25.876723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.876689 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8"] Nov 02 15:48:25.923524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:25.923490 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a2e6f-588fb56b7-mb2n8"] Nov 02 15:48:26.455937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:26.455908 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc2d51c-760f-447d-8a31-675d6dae2424" path="/var/lib/kubelet/pods/9dc2d51c-760f-447d-8a31-675d6dae2424/volumes" Nov 02 15:48:26.475111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:26.475081 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:26.759648 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:26.759551 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:26.912357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:26.912324 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:27.381969 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:27.381936 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:27.611742 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:27.611699 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:27.757043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:27.756940 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:27.963140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:27.963100 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:30.049854 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:30.049817 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:32.192539 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:32.192498 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:33.166335 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:33.166299 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:33.556935 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:33.556862 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:33.698761 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:33.698727 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:33.901864 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:33.901836 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:36.108526 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:36.108491 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:37.386711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:37.386678 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:40.172786 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:40.172748 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv"] Nov 02 15:48:40.335509 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:40.335472 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl"] Nov 02 15:48:40.567234 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:40.567150 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:40.708136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:40.708101 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:41.010992 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:41.010941 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:48:41.956613 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:41.956574 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:42.100209 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:42.100166 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:42.457108 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:42.457073 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:45.173547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.173495 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" podUID="23fdf30f-90a7-47c0-b6e6-6f71caaa8d50" containerName="payload-processing" containerID="cri-o://885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536" gracePeriod=25 Nov 02 15:48:45.336281 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.336244 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" podUID="b524cbab-a209-4ee1-9b40-ed8d682af98b" containerName="payload-pre-processing" containerID="cri-o://45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e" gracePeriod=25 Nov 02 15:48:45.461214 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.461190 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:48:45.514954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.514923 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-plugins-config-volume\") pod \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\" (UID: \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\") " Nov 02 15:48:45.515117 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.515010 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-kube-api-access-6pklq\" (UniqueName: \"kubernetes.io/projected/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-kube-api-access-6pklq\") pod \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\" (UID: \"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50\") " Nov 02 15:48:45.515342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.515290 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-plugins-config-volume" pod "23fdf30f-90a7-47c0-b6e6-6f71caaa8d50" (UID: "23fdf30f-90a7-47c0-b6e6-6f71caaa8d50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:45.515485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.515404 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:45.518159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.518137 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-kube-api-access-6pklq" pod "23fdf30f-90a7-47c0-b6e6-6f71caaa8d50" (UID: "23fdf30f-90a7-47c0-b6e6-6f71caaa8d50"). InnerVolumeSpecName "kube-api-access-6pklq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:45.569134 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.569108 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:48:45.615752 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.615722 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b524cbab-a209-4ee1-9b40-ed8d682af98b-kube-api-access-d4c4v\" (UniqueName: \"kubernetes.io/projected/b524cbab-a209-4ee1-9b40-ed8d682af98b-kube-api-access-d4c4v\") pod \"b524cbab-a209-4ee1-9b40-ed8d682af98b\" (UID: \"b524cbab-a209-4ee1-9b40-ed8d682af98b\") " Nov 02 15:48:45.615949 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.615767 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b524cbab-a209-4ee1-9b40-ed8d682af98b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b524cbab-a209-4ee1-9b40-ed8d682af98b-plugins-config-volume\") pod \"b524cbab-a209-4ee1-9b40-ed8d682af98b\" (UID: \"b524cbab-a209-4ee1-9b40-ed8d682af98b\") " Nov 02 15:48:45.615949 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.615918 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pklq\" (UniqueName: \"kubernetes.io/projected/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50-kube-api-access-6pklq\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:45.616205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.616179 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b524cbab-a209-4ee1-9b40-ed8d682af98b-plugins-config-volume" pod "b524cbab-a209-4ee1-9b40-ed8d682af98b" (UID: "b524cbab-a209-4ee1-9b40-ed8d682af98b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:45.617866 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.617836 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b524cbab-a209-4ee1-9b40-ed8d682af98b-kube-api-access-d4c4v" pod "b524cbab-a209-4ee1-9b40-ed8d682af98b" (UID: "b524cbab-a209-4ee1-9b40-ed8d682af98b"). InnerVolumeSpecName "kube-api-access-d4c4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:45.717003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.716898 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4c4v\" (UniqueName: \"kubernetes.io/projected/b524cbab-a209-4ee1-9b40-ed8d682af98b-kube-api-access-d4c4v\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:45.717003 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.716930 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b524cbab-a209-4ee1-9b40-ed8d682af98b-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:45.815594 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.815557 5433 generic.go:356] "Generic (PLEG): container finished" podID="23fdf30f-90a7-47c0-b6e6-6f71caaa8d50" containerID="885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536" exitCode=0 Nov 02 15:48:45.815797 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.815602 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" Nov 02 15:48:45.815797 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.815653 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" event={"ID":"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50","Type":"ContainerDied","Data":"885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536"} Nov 02 15:48:45.815797 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.815716 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv" event={"ID":"23fdf30f-90a7-47c0-b6e6-6f71caaa8d50","Type":"ContainerDied","Data":"a4fa64532bd9e7186e39001a230b0bbf61e2840b3adc250d516c8dfca9c64232"} Nov 02 15:48:45.815797 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.815780 5433 scope.go:122] "RemoveContainer" containerID="885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536" Nov 02 15:48:45.817136 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.817112 5433 generic.go:356] "Generic (PLEG): container finished" podID="b524cbab-a209-4ee1-9b40-ed8d682af98b" containerID="45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e" exitCode=0 Nov 02 15:48:45.817247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.817160 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" Nov 02 15:48:45.817247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.817159 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" event={"ID":"b524cbab-a209-4ee1-9b40-ed8d682af98b","Type":"ContainerDied","Data":"45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e"} Nov 02 15:48:45.817360 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.817264 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl" event={"ID":"b524cbab-a209-4ee1-9b40-ed8d682af98b","Type":"ContainerDied","Data":"587cda4ad9d87e3a15a992e1b6554fde9b6948943baa57937031c981f6e1f7ec"} Nov 02 15:48:45.825731 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.825630 5433 scope.go:122] "RemoveContainer" containerID="885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536" Nov 02 15:48:45.825939 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:45.825917 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536\": container with ID starting with 885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536 not found: ID does not exist" containerID="885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536" Nov 02 15:48:45.826050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.825946 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536"} err="failed to get container status \"885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536\": rpc error: code = NotFound desc = could not find container \"885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536\": container with ID starting with 885fc4e130e3c87fce384a9e064c4daa2274e4395880953425dd98ad5e008536 not found: ID does not exist" Nov 02 15:48:45.826050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.825962 5433 scope.go:122] "RemoveContainer" containerID="45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e" Nov 02 15:48:45.833851 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.833831 5433 scope.go:122] "RemoveContainer" containerID="45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e" Nov 02 15:48:45.834118 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:48:45.834093 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e\": container with ID starting with 45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e not found: ID does not exist" containerID="45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e" Nov 02 15:48:45.834174 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.834126 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e"} err="failed to get container status \"45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e\": rpc error: code = NotFound desc = could not find container \"45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e\": container with ID starting with 45a6e9ed7acade517f7d231b07223d17efd29467445e789e9486f7774dc1f81e not found: ID does not exist" Nov 02 15:48:45.969757 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.969667 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv"] Nov 02 15:48:45.974524 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:45.974497 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ff6e1f-67bdfccb59-n9rhv"] Nov 02 15:48:46.077591 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:46.077559 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl"] Nov 02 15:48:46.082240 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:46.082211 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ff6e1f-f6c8db84c-gtrhl"] Nov 02 15:48:46.455049 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:46.455012 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fdf30f-90a7-47c0-b6e6-6f71caaa8d50" path="/var/lib/kubelet/pods/23fdf30f-90a7-47c0-b6e6-6f71caaa8d50/volumes" Nov 02 15:48:46.455543 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:46.455522 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b524cbab-a209-4ee1-9b40-ed8d682af98b" path="/var/lib/kubelet/pods/b524cbab-a209-4ee1-9b40-ed8d682af98b/volumes" Nov 02 15:48:47.468481 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:47.468448 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:51.878565 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:51.878519 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:53.530536 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:53.530493 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:53.769803 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:53.769751 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:53.916889 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:53.916851 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:54.485871 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:54.485835 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:48:59.149289 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:48:59.149255 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:00.896424 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:00.896383 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:01.044623 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:01.044589 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:01.221014 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:01.220927 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c"] Nov 02 15:49:01.221234 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:01.221210 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" containerID="cri-o://fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c" gracePeriod=30 Nov 02 15:49:01.229937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:01.229904 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:01.719887 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:01.719849 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:04.532357 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:04.532329 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:06.471777 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.471753 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:49:06.617220 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617158 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-socket\") pod \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " Nov 02 15:49:06.617220 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617213 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-kube-api-access-svwv9\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-kube-api-access-svwv9\") pod \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " Nov 02 15:49:06.617382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617236 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-certs\") pod \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " Nov 02 15:49:06.617382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617274 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-token\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-token\") pod \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " Nov 02 15:49:06.617382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617331 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istiod-ca-cert\") pod \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " Nov 02 15:49:06.617382 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617362 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-podinfo\") pod \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " Nov 02 15:49:06.617595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617410 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-credential-socket\") pod \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " Nov 02 15:49:06.617595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617443 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-socket" pod "9bd210f5-84bd-411e-8efa-8a1ee6f4587b" (UID: "9bd210f5-84bd-411e-8efa-8a1ee6f4587b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:06.617595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617449 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-envoy\") pod \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " Nov 02 15:49:06.617595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617526 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-certs" pod "9bd210f5-84bd-411e-8efa-8a1ee6f4587b" (UID: "9bd210f5-84bd-411e-8efa-8a1ee6f4587b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:06.617595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617531 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-data\") pod \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\" (UID: \"9bd210f5-84bd-411e-8efa-8a1ee6f4587b\") " Nov 02 15:49:06.617869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617708 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-credential-socket" pod "9bd210f5-84bd-411e-8efa-8a1ee6f4587b" (UID: "9bd210f5-84bd-411e-8efa-8a1ee6f4587b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:06.617869 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617757 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istiod-ca-cert" pod "9bd210f5-84bd-411e-8efa-8a1ee6f4587b" (UID: "9bd210f5-84bd-411e-8efa-8a1ee6f4587b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:06.617970 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617888 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-data" pod "9bd210f5-84bd-411e-8efa-8a1ee6f4587b" (UID: "9bd210f5-84bd-411e-8efa-8a1ee6f4587b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:06.618054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.617962 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.618054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.618001 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.618054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.618017 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.618054 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.618031 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.619894 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.619868 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-token" pod "9bd210f5-84bd-411e-8efa-8a1ee6f4587b" (UID: "9bd210f5-84bd-411e-8efa-8a1ee6f4587b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:06.620033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.619894 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-kube-api-access-svwv9" pod "9bd210f5-84bd-411e-8efa-8a1ee6f4587b" (UID: "9bd210f5-84bd-411e-8efa-8a1ee6f4587b"). InnerVolumeSpecName "kube-api-access-svwv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:06.620033 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.619945 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-envoy" pod "9bd210f5-84bd-411e-8efa-8a1ee6f4587b" (UID: "9bd210f5-84bd-411e-8efa-8a1ee6f4587b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:06.620159 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.620033 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-podinfo" pod "9bd210f5-84bd-411e-8efa-8a1ee6f4587b" (UID: "9bd210f5-84bd-411e-8efa-8a1ee6f4587b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:49:06.718710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.718675 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.718710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.718704 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.718710 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.718716 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svwv9\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-kube-api-access-svwv9\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.718957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.718726 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.718957 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.718736 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd210f5-84bd-411e-8efa-8a1ee6f4587b-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.832447 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.832411 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:06.922013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.921951 5433 generic.go:356] "Generic (PLEG): container finished" podID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerID="fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c" exitCode=0 Nov 02 15:49:06.922146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.922017 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" event={"ID":"9bd210f5-84bd-411e-8efa-8a1ee6f4587b","Type":"ContainerDied","Data":"fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c"} Nov 02 15:49:06.922146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.922070 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" event={"ID":"9bd210f5-84bd-411e-8efa-8a1ee6f4587b","Type":"ContainerDied","Data":"0fecb6002d9e52e740d3e2649c88c7e81c21c5fffee84ed8430953f91abdcd70"} Nov 02 15:49:06.922146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.922023 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c" Nov 02 15:49:06.922146 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.922120 5433 scope.go:122] "RemoveContainer" containerID="fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c" Nov 02 15:49:06.931499 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.931479 5433 scope.go:122] "RemoveContainer" containerID="1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c" Nov 02 15:49:06.938994 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.938960 5433 scope.go:122] "RemoveContainer" containerID="fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c" Nov 02 15:49:06.939221 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:49:06.939203 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c\": container with ID starting with fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c not found: ID does not exist" containerID="fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c" Nov 02 15:49:06.939273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.939230 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c"} err="failed to get container status \"fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c\": rpc error: code = NotFound desc = could not find container \"fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c\": container with ID starting with fc053cf9c95fdbc9252aba4ea9ba6e10f3e4c1c44cb688d59eb0823b4a85cc7c not found: ID does not exist" Nov 02 15:49:06.939273 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.939246 5433 scope.go:122] "RemoveContainer" containerID="1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c" Nov 02 15:49:06.939474 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:49:06.939457 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c\": container with ID starting with 1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c not found: ID does not exist" containerID="1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c" Nov 02 15:49:06.939515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:06.939479 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c"} err="failed to get container status \"1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c\": rpc error: code = NotFound desc = could not find container \"1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c\": container with ID starting with 1189a86bb8bac314a202379fd1355432589e77c851006e119455cc3f8ea80d3c not found: ID does not exist" Nov 02 15:49:07.078071 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:07.078043 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c"] Nov 02 15:49:07.123342 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:07.123299 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ff6e1f-openshift-default-c9b75df89-kqk6c"] Nov 02 15:49:08.455073 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:08.455041 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" path="/var/lib/kubelet/pods/9bd210f5-84bd-411e-8efa-8a1ee6f4587b/volumes" Nov 02 15:49:09.656268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:09.656232 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:11.011671 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:49:11.011637 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:49:13.841484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:13.841448 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn"] Nov 02 15:49:14.096694 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:14.096610 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:14.240408 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:14.240372 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:18.842324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:18.842221 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" podUID="5d3aa650-c638-49cf-bcec-6b6201c977ec" containerName="payload-pre-processing" containerID="cri-o://79da29b57a79c048f81e73703333afcd28a7c677041b029a529878969d5cd932" gracePeriod=25 Nov 02 15:49:18.981494 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:18.981465 5433 generic.go:356] "Generic (PLEG): container finished" podID="5d3aa650-c638-49cf-bcec-6b6201c977ec" containerID="79da29b57a79c048f81e73703333afcd28a7c677041b029a529878969d5cd932" exitCode=0 Nov 02 15:49:18.981631 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:18.981519 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" event={"ID":"5d3aa650-c638-49cf-bcec-6b6201c977ec","Type":"ContainerDied","Data":"79da29b57a79c048f81e73703333afcd28a7c677041b029a529878969d5cd932"} Nov 02 15:49:19.086365 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.086339 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:49:19.232183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.232148 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3aa650-c638-49cf-bcec-6b6201c977ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3aa650-c638-49cf-bcec-6b6201c977ec-plugins-config-volume\") pod \"5d3aa650-c638-49cf-bcec-6b6201c977ec\" (UID: \"5d3aa650-c638-49cf-bcec-6b6201c977ec\") " Nov 02 15:49:19.232352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.232199 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3aa650-c638-49cf-bcec-6b6201c977ec-kube-api-access-hfncb\" (UniqueName: \"kubernetes.io/projected/5d3aa650-c638-49cf-bcec-6b6201c977ec-kube-api-access-hfncb\") pod \"5d3aa650-c638-49cf-bcec-6b6201c977ec\" (UID: \"5d3aa650-c638-49cf-bcec-6b6201c977ec\") " Nov 02 15:49:19.232591 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.232571 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3aa650-c638-49cf-bcec-6b6201c977ec-plugins-config-volume" pod "5d3aa650-c638-49cf-bcec-6b6201c977ec" (UID: "5d3aa650-c638-49cf-bcec-6b6201c977ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:19.234276 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.234245 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3aa650-c638-49cf-bcec-6b6201c977ec-kube-api-access-hfncb" pod "5d3aa650-c638-49cf-bcec-6b6201c977ec" (UID: "5d3aa650-c638-49cf-bcec-6b6201c977ec"). InnerVolumeSpecName "kube-api-access-hfncb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:19.333278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.333245 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3aa650-c638-49cf-bcec-6b6201c977ec-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:19.333278 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.333271 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfncb\" (UniqueName: \"kubernetes.io/projected/5d3aa650-c638-49cf-bcec-6b6201c977ec-kube-api-access-hfncb\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:19.987802 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.987768 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" event={"ID":"5d3aa650-c638-49cf-bcec-6b6201c977ec","Type":"ContainerDied","Data":"ae2b5114062dbc058298b3be06e8b15dd568276630d8c627d9fba5c02561c53f"} Nov 02 15:49:19.988290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.987813 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn" Nov 02 15:49:19.988290 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:19.987829 5433 scope.go:122] "RemoveContainer" containerID="79da29b57a79c048f81e73703333afcd28a7c677041b029a529878969d5cd932" Nov 02 15:49:20.150432 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:20.150399 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn"] Nov 02 15:49:20.196916 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:20.196879 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fce776-8f756ddc5-6kgvn"] Nov 02 15:49:20.454837 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:20.454802 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3aa650-c638-49cf-bcec-6b6201c977ec" path="/var/lib/kubelet/pods/5d3aa650-c638-49cf-bcec-6b6201c977ec/volumes" Nov 02 15:49:29.296161 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.296081 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc"] Nov 02 15:49:29.296620 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.296281 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" podUID="424e9afc-422a-4b4d-a56d-af2a9973e329" containerName="maas-api" containerID="cri-o://86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc" gracePeriod=30 Nov 02 15:49:29.438784 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.438749 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:29.557012 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.556924 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:49:29.628130 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.628095 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-metrics-certs\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-metrics-certs\") pod \"424e9afc-422a-4b4d-a56d-af2a9973e329\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " Nov 02 15:49:29.628328 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.628191 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424e9afc-422a-4b4d-a56d-af2a9973e329-kube-api-access-rjngb\" (UniqueName: \"kubernetes.io/projected/424e9afc-422a-4b4d-a56d-af2a9973e329-kube-api-access-rjngb\") pod \"424e9afc-422a-4b4d-a56d-af2a9973e329\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " Nov 02 15:49:29.628328 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.628221 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424e9afc-422a-4b4d-a56d-af2a9973e329-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424e9afc-422a-4b4d-a56d-af2a9973e329-openshift-service-ca\") pod \"424e9afc-422a-4b4d-a56d-af2a9973e329\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " Nov 02 15:49:29.628328 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.628248 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls\") pod \"424e9afc-422a-4b4d-a56d-af2a9973e329\" (UID: \"424e9afc-422a-4b4d-a56d-af2a9973e329\") " Nov 02 15:49:29.628601 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.628577 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424e9afc-422a-4b4d-a56d-af2a9973e329-openshift-service-ca" pod "424e9afc-422a-4b4d-a56d-af2a9973e329" (UID: "424e9afc-422a-4b4d-a56d-af2a9973e329"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:29.630119 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.630095 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-metrics-certs" pod "424e9afc-422a-4b4d-a56d-af2a9973e329" (UID: "424e9afc-422a-4b4d-a56d-af2a9973e329"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:29.630280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.630248 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls" pod "424e9afc-422a-4b4d-a56d-af2a9973e329" (UID: "424e9afc-422a-4b4d-a56d-af2a9973e329"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:29.630368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.630345 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424e9afc-422a-4b4d-a56d-af2a9973e329-kube-api-access-rjngb" pod "424e9afc-422a-4b4d-a56d-af2a9973e329" (UID: "424e9afc-422a-4b4d-a56d-af2a9973e329"). InnerVolumeSpecName "kube-api-access-rjngb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:29.729604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.729569 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjngb\" (UniqueName: \"kubernetes.io/projected/424e9afc-422a-4b4d-a56d-af2a9973e329-kube-api-access-rjngb\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:29.729604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.729599 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424e9afc-422a-4b4d-a56d-af2a9973e329-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:29.729604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.729609 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:29.729851 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:29.729620 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424e9afc-422a-4b4d-a56d-af2a9973e329-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:30.034187 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.034153 5433 generic.go:356] "Generic (PLEG): container finished" podID="424e9afc-422a-4b4d-a56d-af2a9973e329" containerID="86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc" exitCode=0 Nov 02 15:49:30.034403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.034202 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" Nov 02 15:49:30.034403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.034245 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" event={"ID":"424e9afc-422a-4b4d-a56d-af2a9973e329","Type":"ContainerDied","Data":"86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc"} Nov 02 15:49:30.034403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.034298 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc" event={"ID":"424e9afc-422a-4b4d-a56d-af2a9973e329","Type":"ContainerDied","Data":"3dbffbe61e4523497c2781cf186ecbd807dbd7cb840d13b35fc36e6173c156ba"} Nov 02 15:49:30.034403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.034320 5433 scope.go:122] "RemoveContainer" containerID="86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc" Nov 02 15:49:30.042914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.042900 5433 scope.go:122] "RemoveContainer" containerID="86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc" Nov 02 15:49:30.043196 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:49:30.043179 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc\": container with ID starting with 86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc not found: ID does not exist" containerID="86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc" Nov 02 15:49:30.043243 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.043206 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc"} err="failed to get container status \"86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc\": rpc error: code = NotFound desc = could not find container \"86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc\": container with ID starting with 86a62441af4052121967e2bbfb39ccff1dc66be38d034d02354dbbf375ae28fc not found: ID does not exist" Nov 02 15:49:30.189762 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.189721 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc"] Nov 02 15:49:30.191902 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.191872 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fce776-b7b69d65-624fc"] Nov 02 15:49:30.455204 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:30.455172 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424e9afc-422a-4b4d-a56d-af2a9973e329" path="/var/lib/kubelet/pods/424e9afc-422a-4b4d-a56d-af2a9973e329/volumes" Nov 02 15:49:31.045493 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:31.045455 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:31.335515 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:31.335388 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:31.973453 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:31.973393 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:33.471939 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:33.471901 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:35.287307 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:35.287276 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:35.435690 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:35.435658 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:38.367148 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:38.367113 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:38.506667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:38.506640 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:38.719323 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:38.719285 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:41.011232 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:49:41.011194 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:49:43.968217 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:43.968184 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-754d6b6455-5rk2q"] Nov 02 15:49:43.968642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:43.968398 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-754d6b6455-5rk2q" podUID="71a350d7-03c8-45e7-ac2a-de0be80d6855" containerName="manager" containerID="cri-o://85e43fb1fb182e84f4b8ee5a57ee69439456b471a5fc2ffe0ab130f32407ef1d" gracePeriod=30 Nov 02 15:49:44.100433 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.100404 5433 generic.go:356] "Generic (PLEG): container finished" podID="71a350d7-03c8-45e7-ac2a-de0be80d6855" containerID="85e43fb1fb182e84f4b8ee5a57ee69439456b471a5fc2ffe0ab130f32407ef1d" exitCode=0 Nov 02 15:49:44.100572 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.100476 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-5rk2q" event={"ID":"71a350d7-03c8-45e7-ac2a-de0be80d6855","Type":"ContainerDied","Data":"85e43fb1fb182e84f4b8ee5a57ee69439456b471a5fc2ffe0ab130f32407ef1d"} Nov 02 15:49:44.222015 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.221943 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:44.226913 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.226892 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:49:44.260186 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.260143 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-metrics-certs\") pod \"71a350d7-03c8-45e7-ac2a-de0be80d6855\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " Nov 02 15:49:44.260336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.260278 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a350d7-03c8-45e7-ac2a-de0be80d6855-kube-api-access-wd9rh\" (UniqueName: \"kubernetes.io/projected/71a350d7-03c8-45e7-ac2a-de0be80d6855-kube-api-access-wd9rh\") pod \"71a350d7-03c8-45e7-ac2a-de0be80d6855\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " Nov 02 15:49:44.260398 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.260369 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-webhook-cert\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-webhook-cert\") pod \"71a350d7-03c8-45e7-ac2a-de0be80d6855\" (UID: \"71a350d7-03c8-45e7-ac2a-de0be80d6855\") " Nov 02 15:49:44.264805 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.264774 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a350d7-03c8-45e7-ac2a-de0be80d6855-kube-api-access-wd9rh" pod "71a350d7-03c8-45e7-ac2a-de0be80d6855" (UID: "71a350d7-03c8-45e7-ac2a-de0be80d6855"). InnerVolumeSpecName "kube-api-access-wd9rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:44.264998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.264933 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-webhook-cert" pod "71a350d7-03c8-45e7-ac2a-de0be80d6855" (UID: "71a350d7-03c8-45e7-ac2a-de0be80d6855"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:44.265108 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.265083 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-metrics-certs" pod "71a350d7-03c8-45e7-ac2a-de0be80d6855" (UID: "71a350d7-03c8-45e7-ac2a-de0be80d6855"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:44.361612 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.361581 5433 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-webhook-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.361612 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.361612 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a350d7-03c8-45e7-ac2a-de0be80d6855-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.361817 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.361622 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd9rh\" (UniqueName: \"kubernetes.io/projected/71a350d7-03c8-45e7-ac2a-de0be80d6855-kube-api-access-wd9rh\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.565787 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.565701 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:44.718213 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.718180 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:44.967821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:44.967782 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:49:45.106661 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:45.106633 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-5rk2q" Nov 02 15:49:45.107166 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:45.106636 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-5rk2q" event={"ID":"71a350d7-03c8-45e7-ac2a-de0be80d6855","Type":"ContainerDied","Data":"6a90091a9838cb602bbb21c8d4e7eb15488962a5bc017de7fec2d6776d0f2b48"} Nov 02 15:49:45.107166 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:45.106784 5433 scope.go:122] "RemoveContainer" containerID="85e43fb1fb182e84f4b8ee5a57ee69439456b471a5fc2ffe0ab130f32407ef1d" Nov 02 15:49:45.308729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:45.308639 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-754d6b6455-5rk2q"] Nov 02 15:49:45.314430 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:45.314405 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-754d6b6455-5rk2q"] Nov 02 15:49:46.201880 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.201842 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-754d6b6455-qdgdv"] Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202225 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202237 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202248 5433 cpu_manager.go:414] "containerMap: removing container" podUID="9f969b30-d040-4ae8-afcb-3b380e4b14fd" containerName="maas-api" Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202253 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f969b30-d040-4ae8-afcb-3b380e4b14fd" containerName="maas-api" Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202264 5433 cpu_manager.go:414] "containerMap: removing container" podUID="2fad0c18-a050-45bf-be6f-a030762dc6b5" containerName="payload-processing" Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202269 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fad0c18-a050-45bf-be6f-a030762dc6b5" containerName="payload-processing" Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202277 5433 cpu_manager.go:414] "containerMap: removing container" podUID="07fa338b-7a5f-42de-b42b-8037d94048d9" containerName="maas-api" Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202282 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fa338b-7a5f-42de-b42b-8037d94048d9" containerName="maas-api" Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202294 5433 cpu_manager.go:414] "containerMap: removing container" podUID="500c39b4-df23-4fe6-abeb-46b7b9172de2" containerName="payload-processing" Nov 02 15:49:46.202295 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202299 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500c39b4-df23-4fe6-abeb-46b7b9172de2" containerName="payload-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202306 5433 cpu_manager.go:414] "containerMap: removing container" podUID="424e9afc-422a-4b4d-a56d-af2a9973e329" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202310 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424e9afc-422a-4b4d-a56d-af2a9973e329" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202317 5433 cpu_manager.go:414] "containerMap: removing container" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202322 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202329 5433 cpu_manager.go:414] "containerMap: removing container" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202333 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202339 5433 cpu_manager.go:414] "containerMap: removing container" podUID="2b2267b7-0f71-412b-b161-1d2425f884fa" containerName="payload-pre-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202343 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2267b7-0f71-412b-b161-1d2425f884fa" containerName="payload-pre-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202349 5433 cpu_manager.go:414] "containerMap: removing container" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202354 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202361 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c4a51c2d-c28e-4145-91ca-2e3f04f8101c" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202367 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a51c2d-c28e-4145-91ca-2e3f04f8101c" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202373 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b2aa573f-e7f4-4b63-9615-98072075a183" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202377 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2aa573f-e7f4-4b63-9615-98072075a183" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202384 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202388 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202393 5433 cpu_manager.go:414] "containerMap: removing container" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202422 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202430 5433 cpu_manager.go:414] "containerMap: removing container" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202435 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202445 5433 cpu_manager.go:414] "containerMap: removing container" podUID="5d3aa650-c638-49cf-bcec-6b6201c977ec" containerName="payload-pre-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202450 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3aa650-c638-49cf-bcec-6b6201c977ec" containerName="payload-pre-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202456 5433 cpu_manager.go:414] "containerMap: removing container" podUID="440458b0-3f93-4f8a-a9b8-8f0e321bc22c" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202460 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440458b0-3f93-4f8a-a9b8-8f0e321bc22c" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202467 5433 cpu_manager.go:414] "containerMap: removing container" podUID="23fdf30f-90a7-47c0-b6e6-6f71caaa8d50" containerName="payload-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202471 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fdf30f-90a7-47c0-b6e6-6f71caaa8d50" containerName="payload-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202478 5433 cpu_manager.go:414] "containerMap: removing container" podUID="8eccaef4-511c-4792-8a2e-78fd5868d78a" containerName="payload-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202482 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eccaef4-511c-4792-8a2e-78fd5868d78a" containerName="payload-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202489 5433 cpu_manager.go:414] "containerMap: removing container" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202495 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202502 5433 cpu_manager.go:414] "containerMap: removing container" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202506 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202511 5433 cpu_manager.go:414] "containerMap: removing container" podUID="9dc2d51c-760f-447d-8a31-675d6dae2424" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202515 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc2d51c-760f-447d-8a31-675d6dae2424" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202523 5433 cpu_manager.go:414] "containerMap: removing container" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202528 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202536 5433 cpu_manager.go:414] "containerMap: removing container" podUID="12a57e1e-9341-4d1b-98a6-903cc883ecb2" containerName="revoke-keys" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202540 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a57e1e-9341-4d1b-98a6-903cc883ecb2" containerName="revoke-keys" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202546 5433 cpu_manager.go:414] "containerMap: removing container" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202550 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202556 5433 cpu_manager.go:414] "containerMap: removing container" podUID="71a350d7-03c8-45e7-ac2a-de0be80d6855" containerName="manager" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202560 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a350d7-03c8-45e7-ac2a-de0be80d6855" containerName="manager" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202569 5433 cpu_manager.go:414] "containerMap: removing container" podUID="373b0af7-e073-474c-b465-8c0b22b359af" containerName="payload-pre-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202574 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373b0af7-e073-474c-b465-8c0b22b359af" containerName="payload-pre-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202584 5433 cpu_manager.go:414] "containerMap: removing container" podUID="6fc8b9d9-3252-48b6-9644-3107f357b196" containerName="payload-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202588 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc8b9d9-3252-48b6-9644-3107f357b196" containerName="payload-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202595 5433 cpu_manager.go:414] "containerMap: removing container" podUID="69375a37-c259-47f4-a4d7-f45a25a6e85b" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202600 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69375a37-c259-47f4-a4d7-f45a25a6e85b" containerName="maas-api" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202607 5433 cpu_manager.go:414] "containerMap: removing container" podUID="bea5c62e-aa29-4f83-8ca9-abb02750fcfa" containerName="payload-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202611 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bea5c62e-aa29-4f83-8ca9-abb02750fcfa" containerName="payload-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202616 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b524cbab-a209-4ee1-9b40-ed8d682af98b" containerName="payload-pre-processing" Nov 02 15:49:46.202651 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202621 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b524cbab-a209-4ee1-9b40-ed8d682af98b" containerName="payload-pre-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202685 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eccaef4-511c-4792-8a2e-78fd5868d78a" containerName="payload-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202695 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202702 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="435423ae-48db-4912-9d55-5bc14cbd7fbf" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202709 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc2d51c-760f-447d-8a31-675d6dae2424" containerName="maas-api" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202715 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd210f5-84bd-411e-8efa-8a1ee6f4587b" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202720 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202725 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202730 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a57e1e-9341-4d1b-98a6-903cc883ecb2" containerName="revoke-keys" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202735 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5f6ccc-64fe-4d23-b350-ea81c5ea7ba9" containerName="main" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202740 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a350d7-03c8-45e7-ac2a-de0be80d6855" containerName="manager" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202745 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="373b0af7-e073-474c-b465-8c0b22b359af" containerName="payload-pre-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202752 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc8b9d9-3252-48b6-9644-3107f357b196" containerName="payload-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202757 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="69375a37-c259-47f4-a4d7-f45a25a6e85b" containerName="maas-api" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202763 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="bea5c62e-aa29-4f83-8ca9-abb02750fcfa" containerName="payload-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202768 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b524cbab-a209-4ee1-9b40-ed8d682af98b" containerName="payload-pre-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202773 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f969b30-d040-4ae8-afcb-3b380e4b14fd" containerName="maas-api" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202778 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fad0c18-a050-45bf-be6f-a030762dc6b5" containerName="payload-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202783 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fa338b-7a5f-42de-b42b-8037d94048d9" containerName="maas-api" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202788 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="500c39b4-df23-4fe6-abeb-46b7b9172de2" containerName="payload-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202792 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="424e9afc-422a-4b4d-a56d-af2a9973e329" containerName="maas-api" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202797 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d72143-9cba-4630-97c1-094c28c4e01c" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202803 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee74c4b-83fe-42f7-980f-7fc179e9c569" containerName="main" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202808 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2267b7-0f71-412b-b161-1d2425f884fa" containerName="payload-pre-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202812 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3940e38e-34c7-49dd-b315-8a03373d0f59" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202817 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a51c2d-c28e-4145-91ca-2e3f04f8101c" containerName="maas-api" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202822 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2aa573f-e7f4-4b63-9615-98072075a183" containerName="maas-api" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202827 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202832 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54fee09-3c5a-4601-837d-5ca649fdbee9" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202837 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e5919a-b07b-4163-b8fa-910fd2352f06" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202842 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3aa650-c638-49cf-bcec-6b6201c977ec" containerName="payload-pre-processing" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202847 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="440458b0-3f93-4f8a-a9b8-8f0e321bc22c" containerName="maas-api" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202852 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1aacac-a3f6-4b64-84ed-2c673e7edd59" containerName="istio-proxy" Nov 02 15:49:46.204410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.202858 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fdf30f-90a7-47c0-b6e6-6f71caaa8d50" containerName="payload-processing" Nov 02 15:49:46.205870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.205853 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.255696 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.255668 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 15:49:46.255865 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.255704 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 15:49:46.285097 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.285071 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-webhook-cert\") pod \"maas-controller-754d6b6455-qdgdv\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.285247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.285107 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7q84\" (UniqueName: \"kubernetes.io/projected/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-kube-api-access-c7q84\") pod \"maas-controller-754d6b6455-qdgdv\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.285247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.285141 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-metrics-certs\") pod \"maas-controller-754d6b6455-qdgdv\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.302316 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.302292 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8b769\"" Nov 02 15:49:46.302316 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.302297 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 15:49:46.304168 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.304148 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-754d6b6455-qdgdv"] Nov 02 15:49:46.386069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.386038 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-webhook-cert\") pod \"maas-controller-754d6b6455-qdgdv\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.386069 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.386074 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7q84\" (UniqueName: \"kubernetes.io/projected/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-kube-api-access-c7q84\") pod \"maas-controller-754d6b6455-qdgdv\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.386249 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.386094 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-metrics-certs\") pod \"maas-controller-754d6b6455-qdgdv\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.388529 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.388506 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-webhook-cert\") pod \"maas-controller-754d6b6455-qdgdv\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.388642 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.388562 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-metrics-certs\") pod \"maas-controller-754d6b6455-qdgdv\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.444051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.444019 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7q84\" (UniqueName: \"kubernetes.io/projected/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-kube-api-access-c7q84\") pod \"maas-controller-754d6b6455-qdgdv\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.455368 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.455304 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a350d7-03c8-45e7-ac2a-de0be80d6855" path="/var/lib/kubelet/pods/71a350d7-03c8-45e7-ac2a-de0be80d6855/volumes" Nov 02 15:49:46.518727 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.518699 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:46.884542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.884514 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-754d6b6455-qdgdv"] Nov 02 15:49:46.886658 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:49:46.886626 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee86a1d2_2e1d_4201_baf4_89b3a9f34abe.slice/crio-0d74de1e2a7986136d5bbb399e7db68f99e636d3b7f53313a687f3c02ad27d84 WatchSource:0}: Error finding container 0d74de1e2a7986136d5bbb399e7db68f99e636d3b7f53313a687f3c02ad27d84: Status 404 returned error can't find the container with id 0d74de1e2a7986136d5bbb399e7db68f99e636d3b7f53313a687f3c02ad27d84 Nov 02 15:49:46.888018 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:46.887999 5433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:49:47.117958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:47.117914 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-qdgdv" event={"ID":"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe","Type":"ContainerStarted","Data":"0d74de1e2a7986136d5bbb399e7db68f99e636d3b7f53313a687f3c02ad27d84"} Nov 02 15:49:49.128935 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:49.128899 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-qdgdv" event={"ID":"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe","Type":"ContainerStarted","Data":"cc2a03a249f6ced755da35ed40a41fd1459d39b724e3e2967af624be8ed9217f"} Nov 02 15:49:49.129339 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:49.129015 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:49:49.241428 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:49.241375 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-754d6b6455-qdgdv" podStartSLOduration=1.600531755 podStartE2EDuration="3.24136096s" totalImagesPullingTime="1.640829205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:49.240747067 +0000 UTC m=+3092.913950263" watchObservedRunningTime="2026-11-02 15:49:49.24136096 +0000 UTC m=+3092.914564156" Nov 02 15:49:50.845834 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:49:50.845797 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:00.136566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:00.136533 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:50:06.837283 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:06.837247 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:11.011336 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:50:11.011279 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:50:19.991870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:19.991830 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:20.110811 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.110776 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-754d6b6455-qdgdv"] Nov 02 15:50:20.111051 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.111000 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-754d6b6455-qdgdv" podUID="ee86a1d2-2e1d-4201-baf4-89b3a9f34abe" containerName="manager" containerID="cri-o://cc2a03a249f6ced755da35ed40a41fd1459d39b724e3e2967af624be8ed9217f" gracePeriod=30 Nov 02 15:50:20.276422 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.276347 5433 generic.go:356] "Generic (PLEG): container finished" podID="ee86a1d2-2e1d-4201-baf4-89b3a9f34abe" containerID="cc2a03a249f6ced755da35ed40a41fd1459d39b724e3e2967af624be8ed9217f" exitCode=0 Nov 02 15:50:20.276569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.276420 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-qdgdv" event={"ID":"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe","Type":"ContainerDied","Data":"cc2a03a249f6ced755da35ed40a41fd1459d39b724e3e2967af624be8ed9217f"} Nov 02 15:50:20.854521 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.854494 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:50:20.910149 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.910118 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-webhook-cert\") pod \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " Nov 02 15:50:20.910314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.910169 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-metrics-certs\") pod \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " Nov 02 15:50:20.910314 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.910260 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-kube-api-access-c7q84\" (UniqueName: \"kubernetes.io/projected/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-kube-api-access-c7q84\") pod \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\" (UID: \"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe\") " Nov 02 15:50:20.912225 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.912195 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-webhook-cert" pod "ee86a1d2-2e1d-4201-baf4-89b3a9f34abe" (UID: "ee86a1d2-2e1d-4201-baf4-89b3a9f34abe"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:50:20.912355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.912258 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-metrics-certs" pod "ee86a1d2-2e1d-4201-baf4-89b3a9f34abe" (UID: "ee86a1d2-2e1d-4201-baf4-89b3a9f34abe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:50:20.912355 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:20.912310 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-kube-api-access-c7q84" pod "ee86a1d2-2e1d-4201-baf4-89b3a9f34abe" (UID: "ee86a1d2-2e1d-4201-baf4-89b3a9f34abe"). InnerVolumeSpecName "kube-api-access-c7q84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:50:21.011418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.011320 5433 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-webhook-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:50:21.011418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.011364 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:50:21.011418 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.011380 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7q84\" (UniqueName: \"kubernetes.io/projected/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe-kube-api-access-c7q84\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:50:21.045310 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.045278 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4"] Nov 02 15:50:21.051111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.051046 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38602001-6whp4"] Nov 02 15:50:21.282193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.282104 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-qdgdv" Nov 02 15:50:21.282193 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.282108 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-qdgdv" event={"ID":"ee86a1d2-2e1d-4201-baf4-89b3a9f34abe","Type":"ContainerDied","Data":"0d74de1e2a7986136d5bbb399e7db68f99e636d3b7f53313a687f3c02ad27d84"} Nov 02 15:50:21.282457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.282232 5433 scope.go:122] "RemoveContainer" containerID="cc2a03a249f6ced755da35ed40a41fd1459d39b724e3e2967af624be8ed9217f" Nov 02 15:50:21.437064 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.437022 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-754d6b6455-qdgdv"] Nov 02 15:50:21.483569 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:21.483533 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-754d6b6455-qdgdv"] Nov 02 15:50:22.126795 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.126761 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8"] Nov 02 15:50:22.127200 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.127190 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" Nov 02 15:50:22.127247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.127203 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c978cb20-4d48-48db-a682-f8d9916a460e" containerName="istio-proxy" Nov 02 15:50:22.127247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.127228 5433 cpu_manager.go:414] "containerMap: removing container" podUID="ee86a1d2-2e1d-4201-baf4-89b3a9f34abe" containerName="manager" Nov 02 15:50:22.127247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.127233 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee86a1d2-2e1d-4201-baf4-89b3a9f34abe" containerName="manager" Nov 02 15:50:22.127363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.127249 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" Nov 02 15:50:22.127363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.127255 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" Nov 02 15:50:22.127363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.127306 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4120f89-4395-46f6-a62c-3fb79150b20e" containerName="istio-proxy" Nov 02 15:50:22.127363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.127314 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee86a1d2-2e1d-4201-baf4-89b3a9f34abe" containerName="manager" Nov 02 15:50:22.131634 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.131616 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.142954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.142932 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57"] Nov 02 15:50:22.146180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.146155 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.181498 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.181467 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x6s8k\"" Nov 02 15:50:22.227354 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227317 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.227508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227366 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.227508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227467 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.227637 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227507 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.227637 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227564 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59cdv\" (UniqueName: \"kubernetes.io/projected/6b18468c-caaa-4be9-a134-7da1da1fa1c8-kube-api-access-59cdv\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.227713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227648 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npcz\" (UniqueName: \"kubernetes.io/projected/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-kube-api-access-7npcz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.227713 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227679 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.227790 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227712 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.227954 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227916 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.228083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.227984 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.228083 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.228042 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.228212 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.228095 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.228212 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.228124 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.228327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.228242 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.228327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.228264 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.228327 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.228295 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.228474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.228384 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.228474 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.228412 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.230387 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.230366 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8"] Nov 02 15:50:22.296169 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.296138 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57"] Nov 02 15:50:22.329121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329095 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.329265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329125 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.329265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329150 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.329265 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329256 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.329457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329294 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.329457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329311 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.329457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329334 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59cdv\" (UniqueName: \"kubernetes.io/projected/6b18468c-caaa-4be9-a134-7da1da1fa1c8-kube-api-access-59cdv\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.329457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329372 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npcz\" (UniqueName: \"kubernetes.io/projected/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-kube-api-access-7npcz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.329457 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329413 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.329754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329459 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.329754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329502 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.329754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329534 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.329754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329556 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.329754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329561 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.329754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329631 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.329754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329663 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.329754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329701 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.329754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329726 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.330199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329774 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.330199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329775 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.330199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329799 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.330199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329823 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.330199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329843 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.330199 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.329835 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.330449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.330262 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.330449 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.330438 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.330544 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.330461 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.330604 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.330551 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.332184 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.332160 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.332440 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.332422 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.332668 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.332647 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.332711 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.332647 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.389320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.389254 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.389320 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.389303 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npcz\" (UniqueName: \"kubernetes.io/projected/9f7fd75c-268f-4df0-8eaf-7d9648ed4442-kube-api-access-7npcz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4xdd8\" (UID: \"9f7fd75c-268f-4df0-8eaf-7d9648ed4442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.389534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.389511 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59cdv\" (UniqueName: \"kubernetes.io/projected/6b18468c-caaa-4be9-a134-7da1da1fa1c8-kube-api-access-59cdv\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.389649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.389632 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b18468c-caaa-4be9-a134-7da1da1fa1c8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pws57\" (UID: \"6b18468c-caaa-4be9-a134-7da1da1fa1c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.444045 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.444020 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:22.454835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.454807 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a57e1e-9341-4d1b-98a6-903cc883ecb2" path="/var/lib/kubelet/pods/12a57e1e-9341-4d1b-98a6-903cc883ecb2/volumes" Nov 02 15:50:22.455259 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.455240 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee86a1d2-2e1d-4201-baf4-89b3a9f34abe" path="/var/lib/kubelet/pods/ee86a1d2-2e1d-4201-baf4-89b3a9f34abe/volumes" Nov 02 15:50:22.458705 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.458686 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:22.693843 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.693764 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:22.814403 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.814373 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-754d6b6455-xr57v"] Nov 02 15:50:22.818873 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.818852 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:22.861336 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.861310 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8"] Nov 02 15:50:22.863200 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:50:22.863158 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7fd75c_268f_4df0_8eaf_7d9648ed4442.slice/crio-c237f88100f1093158d974bb9aea49d1279d2c11d0b417f94312f535dec23207 WatchSource:0}: Error finding container c237f88100f1093158d974bb9aea49d1279d2c11d0b417f94312f535dec23207: Status 404 returned error can't find the container with id c237f88100f1093158d974bb9aea49d1279d2c11d0b417f94312f535dec23207 Nov 02 15:50:22.865300 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.865266 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:50:22.865400 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.865331 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:50:22.865400 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.865359 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:50:22.869419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.869398 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 15:50:22.869419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.869406 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 15:50:22.869419 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.869395 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8b769\"" Nov 02 15:50:22.905078 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:50:22.905043 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b18468c_caaa_4be9_a134_7da1da1fa1c8.slice/crio-f9c9cf8a408fc20aa5109575741e7ab85cf7bee42ee53e8474e2130620d2b2d2 WatchSource:0}: Error finding container f9c9cf8a408fc20aa5109575741e7ab85cf7bee42ee53e8474e2130620d2b2d2: Status 404 returned error can't find the container with id f9c9cf8a408fc20aa5109575741e7ab85cf7bee42ee53e8474e2130620d2b2d2 Nov 02 15:50:22.906296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.906263 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 15:50:22.907835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.907795 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:50:22.907937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.907828 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-754d6b6455-xr57v"] Nov 02 15:50:22.907937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.907880 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:50:22.907937 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.907927 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:50:22.909082 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.909061 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57"] Nov 02 15:50:22.938410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.935145 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67daa79-d4d3-4f7e-95f0-0815963ecfc0-webhook-cert\") pod \"maas-controller-754d6b6455-xr57v\" (UID: \"d67daa79-d4d3-4f7e-95f0-0815963ecfc0\") " pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:22.938410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.935268 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sgqp\" (UniqueName: \"kubernetes.io/projected/d67daa79-d4d3-4f7e-95f0-0815963ecfc0-kube-api-access-9sgqp\") pod \"maas-controller-754d6b6455-xr57v\" (UID: \"d67daa79-d4d3-4f7e-95f0-0815963ecfc0\") " pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:22.938410 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:22.935432 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67daa79-d4d3-4f7e-95f0-0815963ecfc0-metrics-certs\") pod \"maas-controller-754d6b6455-xr57v\" (UID: \"d67daa79-d4d3-4f7e-95f0-0815963ecfc0\") " pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:23.038087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.037667 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgqp\" (UniqueName: \"kubernetes.io/projected/d67daa79-d4d3-4f7e-95f0-0815963ecfc0-kube-api-access-9sgqp\") pod \"maas-controller-754d6b6455-xr57v\" (UID: \"d67daa79-d4d3-4f7e-95f0-0815963ecfc0\") " pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:23.038087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.037734 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67daa79-d4d3-4f7e-95f0-0815963ecfc0-metrics-certs\") pod \"maas-controller-754d6b6455-xr57v\" (UID: \"d67daa79-d4d3-4f7e-95f0-0815963ecfc0\") " pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:23.038087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.037802 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67daa79-d4d3-4f7e-95f0-0815963ecfc0-webhook-cert\") pod \"maas-controller-754d6b6455-xr57v\" (UID: \"d67daa79-d4d3-4f7e-95f0-0815963ecfc0\") " pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:23.040205 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.040181 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d67daa79-d4d3-4f7e-95f0-0815963ecfc0-metrics-certs\") pod \"maas-controller-754d6b6455-xr57v\" (UID: \"d67daa79-d4d3-4f7e-95f0-0815963ecfc0\") " pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:23.040547 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.040525 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67daa79-d4d3-4f7e-95f0-0815963ecfc0-webhook-cert\") pod \"maas-controller-754d6b6455-xr57v\" (UID: \"d67daa79-d4d3-4f7e-95f0-0815963ecfc0\") " pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:23.096210 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.096173 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgqp\" (UniqueName: \"kubernetes.io/projected/d67daa79-d4d3-4f7e-95f0-0815963ecfc0-kube-api-access-9sgqp\") pod \"maas-controller-754d6b6455-xr57v\" (UID: \"d67daa79-d4d3-4f7e-95f0-0815963ecfc0\") " pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:23.132884 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.132850 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:23.294599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.294497 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" event={"ID":"9f7fd75c-268f-4df0-8eaf-7d9648ed4442","Type":"ContainerStarted","Data":"880b9340067ac69ecd1adca9ad7c57ca9249f6d50cbf189f628a4fae2ba81d60"} Nov 02 15:50:23.294599 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.294565 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" event={"ID":"9f7fd75c-268f-4df0-8eaf-7d9648ed4442","Type":"ContainerStarted","Data":"c237f88100f1093158d974bb9aea49d1279d2c11d0b417f94312f535dec23207"} Nov 02 15:50:23.296288 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.296259 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" event={"ID":"6b18468c-caaa-4be9-a134-7da1da1fa1c8","Type":"ContainerStarted","Data":"21a0afcec560d838bf8f09899013497e1523cb7e82dc904e48ddfd1f52f5010b"} Nov 02 15:50:23.296411 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.296310 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" event={"ID":"6b18468c-caaa-4be9-a134-7da1da1fa1c8","Type":"ContainerStarted","Data":"f9c9cf8a408fc20aa5109575741e7ab85cf7bee42ee53e8474e2130620d2b2d2"} Nov 02 15:50:23.447296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.447174 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:23.454723 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.454689 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4xdd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:50:23.454968 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.454940 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" podUID="9f7fd75c-268f-4df0-8eaf-7d9648ed4442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 15:50:23.463789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.463563 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:23.463789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.463688 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pws57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 15:50:23.463789 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.463730 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podUID="6b18468c-caaa-4be9-a134-7da1da1fa1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 15:50:23.513562 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.513504 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" podStartSLOduration=1.5134852730000001 podStartE2EDuration="1.513485273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:23.42386036 +0000 UTC m=+3127.097063556" watchObservedRunningTime="2026-11-02 15:50:23.513485273 +0000 UTC m=+3127.186688468" Nov 02 15:50:23.513799 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:23.513732 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-754d6b6455-xr57v"] Nov 02 15:50:24.302592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:24.302552 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-xr57v" event={"ID":"d67daa79-d4d3-4f7e-95f0-0815963ecfc0","Type":"ContainerStarted","Data":"fa8d73bcb0dc94a1faa6346c01e4d89633b0af8eb5d61ed30e36483de66b078a"} Nov 02 15:50:24.448537 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:24.448495 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4xdd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:50:24.448839 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:24.448809 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" podUID="9f7fd75c-268f-4df0-8eaf-7d9648ed4442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 15:50:24.475070 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:24.470085 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pws57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 15:50:24.475070 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:24.470162 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podUID="6b18468c-caaa-4be9-a134-7da1da1fa1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 15:50:25.308907 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:25.308865 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-754d6b6455-xr57v" event={"ID":"d67daa79-d4d3-4f7e-95f0-0815963ecfc0","Type":"ContainerStarted","Data":"70e359c636b6a4510cb8cb7551d1e5bd2c81a0171e7b11102bd9796abdd38e48"} Nov 02 15:50:25.309500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:25.309480 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:25.433000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:25.425960 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podStartSLOduration=3.425940577 podStartE2EDuration="3.425940577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:23.554309184 +0000 UTC m=+3127.227512386" watchObservedRunningTime="2026-11-02 15:50:25.425940577 +0000 UTC m=+3129.099143774" Nov 02 15:50:25.433000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:25.426149 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-754d6b6455-xr57v" podStartSLOduration=2.283272947 podStartE2EDuration="3.426141413s" totalImagesPullingTime="1.142868466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:25.424470028 +0000 UTC m=+3129.097673235" watchObservedRunningTime="2026-11-02 15:50:25.426141413 +0000 UTC m=+3129.099344620" Nov 02 15:50:25.453998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:25.447364 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4xdd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:50:25.453998 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:25.447425 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" podUID="9f7fd75c-268f-4df0-8eaf-7d9648ed4442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 15:50:25.463996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:25.462354 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pws57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 15:50:25.463996 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:25.462411 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podUID="6b18468c-caaa-4be9-a134-7da1da1fa1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 15:50:26.448000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:26.447534 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4xdd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:50:26.448000 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:26.447610 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" podUID="9f7fd75c-268f-4df0-8eaf-7d9648ed4442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 15:50:26.468641 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:26.468586 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pws57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 15:50:26.468820 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:26.468665 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podUID="6b18468c-caaa-4be9-a134-7da1da1fa1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 15:50:27.458001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:27.457137 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4xdd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:50:27.458001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:27.457200 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" podUID="9f7fd75c-268f-4df0-8eaf-7d9648ed4442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 15:50:27.467280 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:27.467239 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pws57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 15:50:27.467472 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:27.467308 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podUID="6b18468c-caaa-4be9-a134-7da1da1fa1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 15:50:28.447999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:28.447404 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4xdd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:50:28.447999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:28.447470 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" podUID="9f7fd75c-268f-4df0-8eaf-7d9648ed4442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 15:50:28.465385 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:28.465351 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pws57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 15:50:28.465918 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:28.465889 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podUID="6b18468c-caaa-4be9-a134-7da1da1fa1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 15:50:29.444484 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:29.444441 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4xdd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:50:29.444709 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:29.444505 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" podUID="9f7fd75c-268f-4df0-8eaf-7d9648ed4442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 15:50:29.460039 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:29.460003 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pws57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 15:50:29.460154 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:29.460067 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podUID="6b18468c-caaa-4be9-a134-7da1da1fa1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 15:50:30.459248 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:30.459208 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pws57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 15:50:30.459691 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:30.459280 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podUID="6b18468c-caaa-4be9-a134-7da1da1fa1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 15:50:31.205140 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:31.205107 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:31.205626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:31.205602 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:31.206835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:31.206816 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" Nov 02 15:50:31.463919 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:31.463831 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:31.464428 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:31.464229 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:31.465334 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:31.465301 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" Nov 02 15:50:33.810562 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:33.810528 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:50:37.341677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:37.341641 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-754d6b6455-xr57v" Nov 02 15:50:41.010570 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:50:41.010531 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:50:51.633527 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:50:51.633488 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:09.962955 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:09.962861 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:10.121367 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:10.121332 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:11.010947 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:51:11.010905 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:51:18.514858 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:18.514822 5433 scope.go:122] "RemoveContainer" containerID="72559dd67a03635e37cd77137a2b89804cbe81c97f80f617ee14257439ae3c1f" Nov 02 15:51:34.144121 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:34.144076 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:51:37.808542 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.808512 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz"] Nov 02 15:51:37.812111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.812091 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.855068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.855048 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx"] Nov 02 15:51:37.858441 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.858426 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.880379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880352 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.880480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880389 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7btt\" (UniqueName: \"kubernetes.io/projected/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-kube-api-access-c7btt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.880480 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880420 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.880556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880511 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.880556 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880539 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.880639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880557 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.880639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880577 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.880639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880630 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.880774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880651 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxggk\" (UniqueName: \"kubernetes.io/projected/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-kube-api-access-rxggk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.880774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880666 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.880774 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880745 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.880914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880779 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.880914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880806 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.880914 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880820 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.881095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.880942 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.881095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.881015 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.881095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.881048 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.881095 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.881069 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.909043 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.908918 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz"] Nov 02 15:51:37.954725 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.954686 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx"] Nov 02 15:51:37.981677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.981646 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxggk\" (UniqueName: \"kubernetes.io/projected/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-kube-api-access-rxggk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.981677 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.981678 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.981848 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.981705 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.981909 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.981888 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.981965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.981950 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.982038 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982007 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.982089 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982049 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.982153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982093 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.982153 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982122 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.982261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982149 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.982261 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982151 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.982476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982452 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.982476 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982467 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.982640 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982460 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.982640 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982569 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7btt\" (UniqueName: \"kubernetes.io/projected/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-kube-api-access-c7btt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.982754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982651 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.982754 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982717 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.982870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982745 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.982870 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982601 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.983005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982887 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.983005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982919 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.983005 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.982946 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.983176 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.983002 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.983176 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.983069 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.983284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.983224 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.983284 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.983003 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.983684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.983656 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.983684 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.983672 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.984379 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.984350 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:37.984643 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.984621 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.984780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.984763 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:37.985340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:37.985318 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:38.051985 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.051948 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7btt\" (UniqueName: \"kubernetes.io/projected/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-kube-api-access-c7btt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:38.057729 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.057700 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:38.062172 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.062119 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxggk\" (UniqueName: \"kubernetes.io/projected/2b3ce253-c2a1-4844-a2f5-2b5932a1594d-kube-api-access-rxggk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wlmbx\" (UID: \"2b3ce253-c2a1-4844-a2f5-2b5932a1594d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:38.070794 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.070771 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac33501-52e3-44da-a6d7-4c8f3aa5a0a1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lcwhz\" (UID: \"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:38.124593 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.124556 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:38.173780 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.173184 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:38.538466 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.538438 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz"] Nov 02 15:51:38.541326 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:51:38.541299 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac33501_52e3_44da_a6d7_4c8f3aa5a0a1.slice/crio-d31efcbf14a171894b7e08616b85d34b11d15c08e8785276e59016b49d3395af WatchSource:0}: Error finding container d31efcbf14a171894b7e08616b85d34b11d15c08e8785276e59016b49d3395af: Status 404 returned error can't find the container with id d31efcbf14a171894b7e08616b85d34b11d15c08e8785276e59016b49d3395af Nov 02 15:51:38.543703 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.543670 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.543803 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.543735 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.543803 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.543764 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.589482 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.589412 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx"] Nov 02 15:51:38.591930 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:51:38.591897 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3ce253_c2a1_4844_a2f5_2b5932a1594d.slice/crio-6cbe6280d8d638877d275a315710730a3e763f2d1a2fe5eab7ba5f0851b910cd WatchSource:0}: Error finding container 6cbe6280d8d638877d275a315710730a3e763f2d1a2fe5eab7ba5f0851b910cd: Status 404 returned error can't find the container with id 6cbe6280d8d638877d275a315710730a3e763f2d1a2fe5eab7ba5f0851b910cd Nov 02 15:51:38.594689 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.594658 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.594800 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.594734 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.594800 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.594779 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.678668 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.678630 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" event={"ID":"2b3ce253-c2a1-4844-a2f5-2b5932a1594d","Type":"ContainerStarted","Data":"cc2a705fde91f3f496323ba6c35da3b7ab911694e65864992a797add020256b5"} Nov 02 15:51:38.678832 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.678687 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" event={"ID":"2b3ce253-c2a1-4844-a2f5-2b5932a1594d","Type":"ContainerStarted","Data":"6cbe6280d8d638877d275a315710730a3e763f2d1a2fe5eab7ba5f0851b910cd"} Nov 02 15:51:38.681197 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.681169 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" event={"ID":"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1","Type":"ContainerStarted","Data":"a1f53ba4acf4071a2891bbe1561b79076cc5ebf6a43e8626e35eb3cffa8b53df"} Nov 02 15:51:38.681325 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.681217 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" event={"ID":"bac33501-52e3-44da-a6d7-4c8f3aa5a0a1","Type":"ContainerStarted","Data":"d31efcbf14a171894b7e08616b85d34b11d15c08e8785276e59016b49d3395af"} Nov 02 15:51:38.795180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.795087 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" podStartSLOduration=1.795072223 podStartE2EDuration="1.795072223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:51:38.793808989 +0000 UTC m=+3202.467012184" watchObservedRunningTime="2026-11-02 15:51:38.795072223 +0000 UTC m=+3202.468275417" Nov 02 15:51:38.910566 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:38.910510 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" podStartSLOduration=1.910496422 podStartE2EDuration="1.910496422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:51:38.909221885 +0000 UTC m=+3202.582425110" watchObservedRunningTime="2026-11-02 15:51:38.910496422 +0000 UTC m=+3202.583699617" Nov 02 15:51:39.127195 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:39.127114 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:39.127890 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:39.127550 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lcwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 15:51:39.127890 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:39.127609 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" podUID="bac33501-52e3-44da-a6d7-4c8f3aa5a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 15:51:39.176242 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:39.176072 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:39.176242 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:39.176126 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wlmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 15:51:39.176242 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:39.176201 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" podUID="2b3ce253-c2a1-4844-a2f5-2b5932a1594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 15:51:40.146999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:40.130085 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lcwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 15:51:40.146999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:40.130142 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" podUID="bac33501-52e3-44da-a6d7-4c8f3aa5a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 15:51:40.174044 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:40.174006 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wlmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 15:51:40.174258 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:40.174055 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" podUID="2b3ce253-c2a1-4844-a2f5-2b5932a1594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 15:51:41.009546 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:51:41.009514 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:51:41.139999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:41.139087 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lcwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 15:51:41.139999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:41.139171 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" podUID="bac33501-52e3-44da-a6d7-4c8f3aa5a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 15:51:41.173490 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:41.173451 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wlmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 15:51:41.173958 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:41.173504 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" podUID="2b3ce253-c2a1-4844-a2f5-2b5932a1594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 15:51:42.127438 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:42.127395 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lcwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 15:51:42.127760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:42.127726 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" podUID="bac33501-52e3-44da-a6d7-4c8f3aa5a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 15:51:42.174856 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:42.174824 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wlmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 15:51:42.175347 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:42.175320 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" podUID="2b3ce253-c2a1-4844-a2f5-2b5932a1594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 15:51:43.137001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:43.136879 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lcwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 15:51:43.137001 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:43.136957 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" podUID="bac33501-52e3-44da-a6d7-4c8f3aa5a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 15:51:43.173330 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:43.173289 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wlmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 15:51:43.173521 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:43.173343 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" podUID="2b3ce253-c2a1-4844-a2f5-2b5932a1594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 15:51:44.131999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:44.126379 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lcwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 15:51:44.131999 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:44.126445 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" podUID="bac33501-52e3-44da-a6d7-4c8f3aa5a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 15:51:44.173581 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:44.173551 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wlmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 15:51:44.173838 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:44.173811 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" podUID="2b3ce253-c2a1-4844-a2f5-2b5932a1594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 15:51:45.125454 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:45.125414 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lcwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 02 15:51:45.125668 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:45.125483 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" podUID="bac33501-52e3-44da-a6d7-4c8f3aa5a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 02 15:51:45.173830 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:45.173792 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wlmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 15:51:45.174251 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:45.173863 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" podUID="2b3ce253-c2a1-4844-a2f5-2b5932a1594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 15:51:46.899738 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:46.899699 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:46.901074 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:46.901045 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:46.902132 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:46.902111 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" Nov 02 15:51:46.951216 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:46.951184 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:46.951693 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:46.951665 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:51:46.952316 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:51:46.952294 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" Nov 02 15:52:11.011079 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:52:11.011037 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:52:34.478068 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:52:34.478031 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:52:41.010874 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:52:41.010835 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:53:11.010865 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:53:11.010826 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:53:34.806065 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:53:34.806016 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:53:41.011277 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:53:41.011237 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:54:07.651822 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:07.651736 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:54:11.011276 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:54:11.011240 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:54:32.252127 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:32.252084 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4xdd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:54:32.252760 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:32.252722 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4xdd8" podUID="9f7fd75c-268f-4df0-8eaf-7d9648ed4442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:54:32.528496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:32.528080 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pws57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:54:32.528496 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:32.528148 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pws57" podUID="6b18468c-caaa-4be9-a134-7da1da1fa1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:54:32.945145 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:32.945098 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lcwhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:54:32.945508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:32.945467 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lcwhz" podUID="bac33501-52e3-44da-a6d7-4c8f3aa5a0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:54:33.002112 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:33.002073 5433 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wlmbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:54:33.002420 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:33.002388 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wlmbx" podUID="2b3ce253-c2a1-4844-a2f5-2b5932a1594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:54:35.135452 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:54:35.135417 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:54:41.010337 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:54:41.010301 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:55:11.010947 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:55:11.010899 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:55:35.505951 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:55:35.505914 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:55:41.181011 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:55:41.180950 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:56:11.011528 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:56:11.011489 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:56:32.914180 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:32.914143 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:33.517364 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:33.517310 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:33.722302 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:33.722260 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:35.809643 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:35.809605 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:37.335579 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:37.335543 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:37.517309 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:37.517271 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:37.708052 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:37.708003 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:40.517831 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:40.517752 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:40.697965 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:40.697923 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:40.930769 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:40.930735 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr"] Nov 02 15:56:40.934370 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:40.934347 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:40.983663 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:40.983631 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a4b243\"" Nov 02 15:56:41.017018 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:56:41.014521 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:56:41.032050 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.032020 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr"] Nov 02 15:56:41.032233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.032111 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5b60b7-221d-4e9b-8b36-8884b4138cec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr\" (UID: \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:41.032233 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.032223 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54j67\" (UniqueName: \"kubernetes.io/projected/3f5b60b7-221d-4e9b-8b36-8884b4138cec-kube-api-access-54j67\") pod \"payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr\" (UID: \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:41.095639 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.095598 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf"] Nov 02 15:56:41.100141 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.100099 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:41.133667 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.133633 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5b60b7-221d-4e9b-8b36-8884b4138cec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr\" (UID: \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:41.133834 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.133718 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54j67\" (UniqueName: \"kubernetes.io/projected/3f5b60b7-221d-4e9b-8b36-8884b4138cec-kube-api-access-54j67\") pod \"payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr\" (UID: \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:41.134340 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.134321 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5b60b7-221d-4e9b-8b36-8884b4138cec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr\" (UID: \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:41.149821 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.149785 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7a0a0a59-dockercfg-qkqk2\"" Nov 02 15:56:41.150508 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.150466 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf"] Nov 02 15:56:41.191688 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.191611 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54j67\" (UniqueName: \"kubernetes.io/projected/3f5b60b7-221d-4e9b-8b36-8884b4138cec-kube-api-access-54j67\") pod \"payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr\" (UID: \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:41.234416 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.234382 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf\" (UID: \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:41.234592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.234465 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9pw2\" (UniqueName: \"kubernetes.io/projected/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-kube-api-access-r9pw2\") pod \"payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf\" (UID: \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:41.251344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.251313 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:41.335485 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.335450 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf\" (UID: \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:41.335637 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.335553 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9pw2\" (UniqueName: \"kubernetes.io/projected/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-kube-api-access-r9pw2\") pod \"payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf\" (UID: \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:41.336188 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.336167 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf\" (UID: \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:41.394239 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.394212 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9pw2\" (UniqueName: \"kubernetes.io/projected/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-kube-api-access-r9pw2\") pod \"payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf\" (UID: \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:41.415088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.415058 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:41.614471 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.614445 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr"] Nov 02 15:56:41.616516 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:56:41.616481 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5b60b7_221d_4e9b_8b36_8884b4138cec.slice/crio-98258081d44c1b7ab6d4e035c0308a6cc5466b10f59e5ed61411fb101ef1e70b WatchSource:0}: Error finding container 98258081d44c1b7ab6d4e035c0308a6cc5466b10f59e5ed61411fb101ef1e70b: Status 404 returned error can't find the container with id 98258081d44c1b7ab6d4e035c0308a6cc5466b10f59e5ed61411fb101ef1e70b Nov 02 15:56:41.782444 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:41.782416 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf"] Nov 02 15:56:41.784392 ip-10-0-135-181 kubenswrapper[5433]: W1102 15:56:41.784368 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e3a8c6_ca7f_4246_bb4f_528cd0eb2ed9.slice/crio-5c8df43fd2bd4780681648b37eb6b741d22bfa545604f2c3795f6a7fd92aceb3 WatchSource:0}: Error finding container 5c8df43fd2bd4780681648b37eb6b741d22bfa545604f2c3795f6a7fd92aceb3: Status 404 returned error can't find the container with id 5c8df43fd2bd4780681648b37eb6b741d22bfa545604f2c3795f6a7fd92aceb3 Nov 02 15:56:42.349535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:42.349501 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" event={"ID":"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9","Type":"ContainerStarted","Data":"28935b32503ba966d2b36fb43f6cf08f24681559f4b6542d18fc45a03b23e7aa"} Nov 02 15:56:42.349743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:42.349547 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" event={"ID":"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9","Type":"ContainerStarted","Data":"5c8df43fd2bd4780681648b37eb6b741d22bfa545604f2c3795f6a7fd92aceb3"} Nov 02 15:56:42.349743 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:42.349587 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:42.351167 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:42.351146 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" event={"ID":"3f5b60b7-221d-4e9b-8b36-8884b4138cec","Type":"ContainerStarted","Data":"474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7"} Nov 02 15:56:42.351268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:42.351177 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" event={"ID":"3f5b60b7-221d-4e9b-8b36-8884b4138cec","Type":"ContainerStarted","Data":"98258081d44c1b7ab6d4e035c0308a6cc5466b10f59e5ed61411fb101ef1e70b"} Nov 02 15:56:42.351268 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:42.351206 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:42.461081 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:42.461035 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" podStartSLOduration=1.4610205330000001 podStartE2EDuration="1.461020533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:56:42.459396331 +0000 UTC m=+3506.132599526" watchObservedRunningTime="2026-11-02 15:56:42.461020533 +0000 UTC m=+3506.134223727" Nov 02 15:56:42.571026 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:42.569134 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" podStartSLOduration=2.5691159580000003 podStartE2EDuration="2.569115958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:56:42.568699505 +0000 UTC m=+3506.241902696" watchObservedRunningTime="2026-11-02 15:56:42.569115958 +0000 UTC m=+3506.242319195" Nov 02 15:56:43.562211 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:43.562175 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:43.713951 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:43.713907 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:43.858277 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:43.858172 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:47.402682 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:47.402646 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:48.526874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:48.526839 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:53.335087 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:53.335045 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:56:53.359595 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:53.359568 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:56:53.359783 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:53.359633 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:56:58.039872 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:56:58.039783 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:02.304156 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:02.304117 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:02.870574 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:02.870532 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:03.027946 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:03.027917 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:03.176299 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:03.176262 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:05.826838 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:05.826809 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:07.812369 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:07.812338 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:11.010811 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:57:11.010777 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:57:12.522309 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:12.522273 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:17.220933 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:17.220898 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:22.017196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:22.017158 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:26.664344 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:26.664303 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:31.326626 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:31.326588 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:36.017032 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:36.016989 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:36.177063 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:36.177027 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:40.822124 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:40.822082 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:41.011967 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:57:41.011934 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:57:45.521479 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:45.521436 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:50.233622 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:50.233585 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:55.000297 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:55.000262 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:57:59.722323 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:57:59.722288 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:01.789473 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:01.789437 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:04.722235 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:04.722199 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:06.829088 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:06.829054 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:09.709592 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:09.709558 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:11.011918 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:58:11.011878 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:58:13.769537 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:13.769498 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf"] Nov 02 15:58:13.871352 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:13.871311 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr"] Nov 02 15:58:14.127381 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:14.127348 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:14.280570 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:14.280530 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:14.426182 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:14.426086 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:18.403673 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:18.403634 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:18.770013 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:18.769894 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" podUID="11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" containerName="payload-processing" containerID="cri-o://28935b32503ba966d2b36fb43f6cf08f24681559f4b6542d18fc45a03b23e7aa" gracePeriod=25 Nov 02 15:58:18.872296 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:18.872255 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" podUID="3f5b60b7-221d-4e9b-8b36-8884b4138cec" containerName="payload-pre-processing" containerID="cri-o://474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7" gracePeriod=25 Nov 02 15:58:19.898431 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:19.898398 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:20.170895 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:20.170810 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:20.925372 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:20.925331 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:23.357535 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:23.357496 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" podUID="11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:58:23.358028 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:23.357497 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" podUID="3f5b60b7-221d-4e9b-8b36-8884b4138cec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:58:27.108827 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:27.108743 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:27.254674 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:27.254635 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:27.436062 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:27.436028 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:33.357649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:33.357611 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" podUID="11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:58:33.357649 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:33.357632 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" podUID="3f5b60b7-221d-4e9b-8b36-8884b4138cec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:58:36.460183 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:36.460121 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:38.912270 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:38.912223 5433 generic.go:356] "Generic (PLEG): container finished" podID="11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" containerID="28935b32503ba966d2b36fb43f6cf08f24681559f4b6542d18fc45a03b23e7aa" exitCode=0 Nov 02 15:58:38.912633 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:38.912301 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" event={"ID":"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9","Type":"ContainerDied","Data":"28935b32503ba966d2b36fb43f6cf08f24681559f4b6542d18fc45a03b23e7aa"} Nov 02 15:58:39.030534 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.030472 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:58:39.131023 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.130958 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:58:39.144668 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.144646 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-kube-api-access-r9pw2\" (UniqueName: \"kubernetes.io/projected/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-kube-api-access-r9pw2\") pod \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\" (UID: \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\") " Nov 02 15:58:39.144835 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.144809 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-plugins-config-volume\") pod \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\" (UID: \"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9\") " Nov 02 15:58:39.145656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.145630 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-plugins-config-volume" pod "11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" (UID: "11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:58:39.146874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.146847 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-kube-api-access-r9pw2" pod "11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" (UID: "11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9"). InnerVolumeSpecName "kube-api-access-r9pw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:58:39.246111 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.246074 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5b60b7-221d-4e9b-8b36-8884b4138cec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5b60b7-221d-4e9b-8b36-8884b4138cec-plugins-config-volume\") pod \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\" (UID: \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\") " Nov 02 15:58:39.246266 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.246130 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5b60b7-221d-4e9b-8b36-8884b4138cec-kube-api-access-54j67\" (UniqueName: \"kubernetes.io/projected/3f5b60b7-221d-4e9b-8b36-8884b4138cec-kube-api-access-54j67\") pod \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\" (UID: \"3f5b60b7-221d-4e9b-8b36-8884b4138cec\") " Nov 02 15:58:39.246433 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.246419 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:39.246470 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.246434 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9pw2\" (UniqueName: \"kubernetes.io/projected/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9-kube-api-access-r9pw2\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:39.246533 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.246509 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5b60b7-221d-4e9b-8b36-8884b4138cec-plugins-config-volume" pod "3f5b60b7-221d-4e9b-8b36-8884b4138cec" (UID: "3f5b60b7-221d-4e9b-8b36-8884b4138cec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:58:39.248190 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.248173 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5b60b7-221d-4e9b-8b36-8884b4138cec-kube-api-access-54j67" pod "3f5b60b7-221d-4e9b-8b36-8884b4138cec" (UID: "3f5b60b7-221d-4e9b-8b36-8884b4138cec"). InnerVolumeSpecName "kube-api-access-54j67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:58:39.347656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.347576 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5b60b7-221d-4e9b-8b36-8884b4138cec-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:39.347656 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.347603 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54j67\" (UniqueName: \"kubernetes.io/projected/3f5b60b7-221d-4e9b-8b36-8884b4138cec-kube-api-access-54j67\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:39.918500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.918466 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" Nov 02 15:58:39.918993 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.918497 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf" event={"ID":"11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9","Type":"ContainerDied","Data":"5c8df43fd2bd4780681648b37eb6b741d22bfa545604f2c3795f6a7fd92aceb3"} Nov 02 15:58:39.918993 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.918564 5433 scope.go:122] "RemoveContainer" containerID="28935b32503ba966d2b36fb43f6cf08f24681559f4b6542d18fc45a03b23e7aa" Nov 02 15:58:39.919871 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.919841 5433 generic.go:356] "Generic (PLEG): container finished" podID="3f5b60b7-221d-4e9b-8b36-8884b4138cec" containerID="474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7" exitCode=0 Nov 02 15:58:39.919938 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.919891 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" Nov 02 15:58:39.920024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.919931 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" event={"ID":"3f5b60b7-221d-4e9b-8b36-8884b4138cec","Type":"ContainerDied","Data":"474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7"} Nov 02 15:58:39.920024 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.920011 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr" event={"ID":"3f5b60b7-221d-4e9b-8b36-8884b4138cec","Type":"ContainerDied","Data":"98258081d44c1b7ab6d4e035c0308a6cc5466b10f59e5ed61411fb101ef1e70b"} Nov 02 15:58:39.928247 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.928230 5433 scope.go:122] "RemoveContainer" containerID="474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7" Nov 02 15:58:39.935874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.935856 5433 scope.go:122] "RemoveContainer" containerID="474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7" Nov 02 15:58:39.936141 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:58:39.936122 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7\": container with ID starting with 474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7 not found: ID does not exist" containerID="474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7" Nov 02 15:58:39.936196 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:39.936148 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7"} err="failed to get container status \"474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7\": rpc error: code = NotFound desc = could not find container \"474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7\": container with ID starting with 474649122de0af5cb612c1883efceab8b3954f70c5cd77fbaa78ec530cf4d6c7 not found: ID does not exist" Nov 02 15:58:40.076500 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:40.076418 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr"] Nov 02 15:58:40.080944 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:40.080920 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a4b243-68dfdphrkr"] Nov 02 15:58:40.228324 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:40.228291 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf"] Nov 02 15:58:40.233363 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:40.233330 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a4b243-67fc8b669l2zcf"] Nov 02 15:58:40.455010 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:40.454958 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" path="/var/lib/kubelet/pods/11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9/volumes" Nov 02 15:58:40.455473 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:40.455450 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5b60b7-221d-4e9b-8b36-8884b4138cec" path="/var/lib/kubelet/pods/3f5b60b7-221d-4e9b-8b36-8884b4138cec/volumes" Nov 02 15:58:41.011669 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:58:41.011630 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:58:43.253425 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:43.253385 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:44.534874 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:44.534841 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:58:50.383100 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:58:50.383063 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:11.011582 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:59:11.011543 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:59:36.842434 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:59:36.842397 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 15:59:41.011193 ip-10-0-135-181 kubenswrapper[5433]: E1102 15:59:41.011148 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 15:59:56.462756 ip-10-0-135-181 kubenswrapper[5433]: I1102 15:59:56.462671 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:11.011570 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:00:11.011526 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:00:12.861865 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:12.861821 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:13.028480 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:13.028448 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:37.135998 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:37.135116 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:39.463683 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:39.463648 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:39.604132 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:39.604096 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:41.010942 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:00:41.010901 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:00:58.773326 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:58.773289 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:59.338817 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.338781 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg"] Nov 02 16:00:59.339220 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.339206 5433 cpu_manager.go:414] "containerMap: removing container" podUID="3f5b60b7-221d-4e9b-8b36-8884b4138cec" containerName="payload-pre-processing" Nov 02 16:00:59.339280 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.339221 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5b60b7-221d-4e9b-8b36-8884b4138cec" containerName="payload-pre-processing" Nov 02 16:00:59.339280 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.339260 5433 cpu_manager.go:414] "containerMap: removing container" podUID="11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" containerName="payload-processing" Nov 02 16:00:59.339280 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.339267 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" containerName="payload-processing" Nov 02 16:00:59.339392 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.339333 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5b60b7-221d-4e9b-8b36-8884b4138cec" containerName="payload-pre-processing" Nov 02 16:00:59.339392 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.339345 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e3a8c6-ca7f-4246-bb4f-528cd0eb2ed9" containerName="payload-processing" Nov 02 16:00:59.342590 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.342570 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.392441 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.392410 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-1d7f44-openshift-default-dockercfg-scptm\"" Nov 02 16:00:59.407752 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.407722 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-credential-socket\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.407917 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.407771 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-envoy\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.407917 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.407884 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-data\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.408079 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.407917 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-certs\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.408079 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.407938 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-podinfo\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.408079 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.407995 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chfbf\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-kube-api-access-chfbf\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.408079 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.408056 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c5ef7-66b7-41db-ad79-3e917389a96f-istiod-ca-cert\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.408250 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.408123 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-socket\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.408250 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.408162 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-token\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.418089 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.418058 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:59.443616 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.443578 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg"] Nov 02 16:00:59.508822 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.508783 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chfbf\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-kube-api-access-chfbf\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.508822 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.508819 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c5ef7-66b7-41db-ad79-3e917389a96f-istiod-ca-cert\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509116 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.508845 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-socket\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509116 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.508957 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-token\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509116 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509064 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-credential-socket\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509326 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509257 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-socket\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509409 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509384 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-envoy\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509475 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509435 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-data\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509475 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509462 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-credential-socket\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509608 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509480 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-certs\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509608 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509525 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-podinfo\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509725 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509606 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c5ef7-66b7-41db-ad79-3e917389a96f-istiod-ca-cert\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509725 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509711 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-data\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.509828 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.509785 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-certs\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.511487 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.511462 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-envoy\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.511806 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.511787 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-podinfo\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.567104 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.567069 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-token\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.567280 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.567136 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chfbf\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-kube-api-access-chfbf\") pod \"e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:00:59.568468 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.568437 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:00:59.655482 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:00:59.655441 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:01:00.095278 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:00.095217 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg"] Nov 02 16:01:00.097873 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:01:00.097845 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083c5ef7_66b7_41db_ad79_3e917389a96f.slice/crio-fd90e388a20aaa740987a01a2ecbae84766793abc3397154c9415ed280c71e06 WatchSource:0}: Error finding container fd90e388a20aaa740987a01a2ecbae84766793abc3397154c9415ed280c71e06: Status 404 returned error can't find the container with id fd90e388a20aaa740987a01a2ecbae84766793abc3397154c9415ed280c71e06 Nov 02 16:01:00.100082 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:00.100050 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:01:00.100178 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:00.100120 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:01:00.100178 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:00.100149 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:01:00.598011 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:00.597957 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" event={"ID":"083c5ef7-66b7-41db-ad79-3e917389a96f","Type":"ContainerStarted","Data":"496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180"} Nov 02 16:01:00.598196 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:00.598029 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" event={"ID":"083c5ef7-66b7-41db-ad79-3e917389a96f","Type":"ContainerStarted","Data":"fd90e388a20aaa740987a01a2ecbae84766793abc3397154c9415ed280c71e06"} Nov 02 16:01:00.656530 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:00.656499 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:01:00.714663 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:00.714599 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" podStartSLOduration=1.7145831249999999 podStartE2EDuration="1.714583125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:01:00.71281573 +0000 UTC m=+3764.386018921" watchObservedRunningTime="2026-11-02 16:01:00.714583125 +0000 UTC m=+3764.387786321" Nov 02 16:01:01.661259 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:01.661231 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:01:02.607181 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:02.607144 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:01:02.608270 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:02.608247 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:01:03.565185 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:03.565152 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:03.729362 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:03.729321 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:03.873908 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:03.873822 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:06.823952 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:06.823901 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:06.989209 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:06.989177 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:07.365675 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.365641 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7"] Nov 02 16:01:07.371271 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.371016 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:07.412347 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.412315 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wh4\" (UniqueName: \"kubernetes.io/projected/69c42886-45ec-4ff8-849e-124210ee7b01-kube-api-access-68wh4\") pod \"payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7\" (UID: \"69c42886-45ec-4ff8-849e-124210ee7b01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:07.412493 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.412434 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c42886-45ec-4ff8-849e-124210ee7b01-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7\" (UID: \"69c42886-45ec-4ff8-849e-124210ee7b01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:07.421025 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.420992 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7"] Nov 02 16:01:07.466295 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.466264 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-1d7f44\"" Nov 02 16:01:07.466295 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.466280 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-61f60231-dockercfg-l4gnt\"" Nov 02 16:01:07.513656 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.513624 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c42886-45ec-4ff8-849e-124210ee7b01-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7\" (UID: \"69c42886-45ec-4ff8-849e-124210ee7b01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:07.513811 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.513694 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wh4\" (UniqueName: \"kubernetes.io/projected/69c42886-45ec-4ff8-849e-124210ee7b01-kube-api-access-68wh4\") pod \"payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7\" (UID: \"69c42886-45ec-4ff8-849e-124210ee7b01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:07.514352 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.514332 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c42886-45ec-4ff8-849e-124210ee7b01-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7\" (UID: \"69c42886-45ec-4ff8-849e-124210ee7b01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:07.572505 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.572469 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wh4\" (UniqueName: \"kubernetes.io/projected/69c42886-45ec-4ff8-849e-124210ee7b01-kube-api-access-68wh4\") pod \"payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7\" (UID: \"69c42886-45ec-4ff8-849e-124210ee7b01\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:07.710737 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:07.710708 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:08.075036 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:08.074996 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7"] Nov 02 16:01:08.077762 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:01:08.077735 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c42886_45ec_4ff8_849e_124210ee7b01.slice/crio-44232c32fcb5d4c2c7e8e9acde86d032c60b63e908c6e225185345570a6e9f31 WatchSource:0}: Error finding container 44232c32fcb5d4c2c7e8e9acde86d032c60b63e908c6e225185345570a6e9f31: Status 404 returned error can't find the container with id 44232c32fcb5d4c2c7e8e9acde86d032c60b63e908c6e225185345570a6e9f31 Nov 02 16:01:08.635252 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:08.635207 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" event={"ID":"69c42886-45ec-4ff8-849e-124210ee7b01","Type":"ContainerStarted","Data":"59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243"} Nov 02 16:01:08.635458 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:08.635278 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:08.635458 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:08.635300 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" event={"ID":"69c42886-45ec-4ff8-849e-124210ee7b01","Type":"ContainerStarted","Data":"44232c32fcb5d4c2c7e8e9acde86d032c60b63e908c6e225185345570a6e9f31"} Nov 02 16:01:08.746019 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:08.745924 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" podStartSLOduration=1.7459102990000002 podStartE2EDuration="1.745910299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:01:08.745369213 +0000 UTC m=+3772.418572409" watchObservedRunningTime="2026-11-02 16:01:08.745910299 +0000 UTC m=+3772.419113494" Nov 02 16:01:09.690319 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:09.690282 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:10.026418 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:10.026327 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:11.011423 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:01:11.011386 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:01:14.503250 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:14.503212 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:19.173788 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:19.173747 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:19.643160 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:19.643126 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:01:23.914203 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:23.914160 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:27.724251 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:27.724212 5433 generic.go:356] "Generic (PLEG): container finished" podID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerID="496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180" exitCode=0 Nov 02 16:01:27.724805 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:27.724285 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" event={"ID":"083c5ef7-66b7-41db-ad79-3e917389a96f","Type":"ContainerDied","Data":"496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180"} Nov 02 16:01:27.724805 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:27.724738 5433 scope.go:122] "RemoveContainer" containerID="496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180" Nov 02 16:01:27.726151 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:27.726124 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:01:27.726239 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:27.726197 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:01:27.726239 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:27.726236 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:01:28.389368 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:28.389330 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:28.747940 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:28.747849 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" event={"ID":"083c5ef7-66b7-41db-ad79-3e917389a96f","Type":"ContainerStarted","Data":"c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465"} Nov 02 16:01:28.956259 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:28.956213 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:29.163900 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:29.163845 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:29.313985 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:29.313943 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:29.656018 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:29.655969 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:01:29.657292 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:29.657267 5433 patch_prober.go:29] interesting pod/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 02 16:01:29.657431 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:29.657314 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 02 16:01:30.656379 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:30.656344 5433 patch_prober.go:29] interesting pod/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 02 16:01:30.656799 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:30.656398 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 02 16:01:31.656267 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:31.656229 5433 patch_prober.go:29] interesting pod/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 02 16:01:31.656462 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:31.656287 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 02 16:01:32.149760 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:32.149729 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:32.607658 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:32.607556 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:01:32.656113 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:32.656076 5433 patch_prober.go:29] interesting pod/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 02 16:01:32.656271 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:32.656133 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 02 16:01:33.656224 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:33.656184 5433 patch_prober.go:29] interesting pod/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 02 16:01:33.656689 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:33.656241 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 02 16:01:33.884604 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:33.884574 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:34.659909 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:34.659882 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:01:34.661095 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:34.661075 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:01:37.457408 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:37.457374 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:38.767096 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:38.767059 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:41.017873 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:01:41.017829 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:01:43.231429 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:43.231395 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:47.908890 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:47.908853 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:52.519877 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:52.519842 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:01:57.247018 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:01:57.246964 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:00.763066 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:00.763026 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:01.845883 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:01.845848 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:06.513041 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:06.512962 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:11.011020 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:02:11.010964 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:02:11.132554 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:11.132524 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:15.821745 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:15.821708 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:20.438203 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:20.438162 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:25.124085 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:25.124046 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:28.312339 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:28.312301 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:29.841451 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:29.841408 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:33.264497 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:33.264460 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:34.828044 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:34.828008 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:37.791555 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:37.791522 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:39.427682 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:39.427649 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:40.176761 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:40.176723 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7"] Nov 02 16:02:40.529372 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:40.529282 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:40.675588 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:40.675551 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:41.009799 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:02:41.009765 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:02:43.513404 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:43.513373 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:45.002029 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:45.001971 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:45.177382 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:45.177334 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" podUID="69c42886-45ec-4ff8-849e-124210ee7b01" containerName="payload-processing" containerID="cri-o://59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243" gracePeriod=25 Nov 02 16:02:45.295531 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:45.295434 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:46.021003 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:46.020943 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:49.642201 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:49.642167 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" podUID="69c42886-45ec-4ff8-849e-124210ee7b01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 16:02:50.688730 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:50.688697 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:50.908076 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:50.908020 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:51.052653 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:51.052567 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:52.096743 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:52.096685 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:52.250633 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:52.250596 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:52.407855 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:52.407823 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg"] Nov 02 16:02:52.408071 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:52.408050 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" containerID="cri-o://c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465" gracePeriod=30 Nov 02 16:02:52.424184 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:52.424151 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:57.106406 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.106309 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:02:57.660573 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.660546 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:02:57.775392 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775284 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-kube-api-access-chfbf\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-kube-api-access-chfbf\") pod \"083c5ef7-66b7-41db-ad79-3e917389a96f\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " Nov 02 16:02:57.775392 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775338 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-envoy\") pod \"083c5ef7-66b7-41db-ad79-3e917389a96f\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " Nov 02 16:02:57.775392 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775393 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083c5ef7-66b7-41db-ad79-3e917389a96f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c5ef7-66b7-41db-ad79-3e917389a96f-istiod-ca-cert\") pod \"083c5ef7-66b7-41db-ad79-3e917389a96f\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " Nov 02 16:02:57.775664 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775510 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-certs\") pod \"083c5ef7-66b7-41db-ad79-3e917389a96f\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " Nov 02 16:02:57.775664 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775543 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-data\") pod \"083c5ef7-66b7-41db-ad79-3e917389a96f\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " Nov 02 16:02:57.775664 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775582 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-token\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-token\") pod \"083c5ef7-66b7-41db-ad79-3e917389a96f\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " Nov 02 16:02:57.775664 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775618 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-credential-socket\") pod \"083c5ef7-66b7-41db-ad79-3e917389a96f\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " Nov 02 16:02:57.775664 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775663 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-podinfo\") pod \"083c5ef7-66b7-41db-ad79-3e917389a96f\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " Nov 02 16:02:57.776069 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775737 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-socket\") pod \"083c5ef7-66b7-41db-ad79-3e917389a96f\" (UID: \"083c5ef7-66b7-41db-ad79-3e917389a96f\") " Nov 02 16:02:57.776069 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.775807 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-certs" pod "083c5ef7-66b7-41db-ad79-3e917389a96f" (UID: "083c5ef7-66b7-41db-ad79-3e917389a96f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:02:57.776204 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.776169 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:57.776437 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.776371 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-socket" pod "083c5ef7-66b7-41db-ad79-3e917389a96f" (UID: "083c5ef7-66b7-41db-ad79-3e917389a96f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:02:57.776437 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.776391 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-data" pod "083c5ef7-66b7-41db-ad79-3e917389a96f" (UID: "083c5ef7-66b7-41db-ad79-3e917389a96f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:02:57.776658 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.776517 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-credential-socket" pod "083c5ef7-66b7-41db-ad79-3e917389a96f" (UID: "083c5ef7-66b7-41db-ad79-3e917389a96f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:02:57.776658 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.776650 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083c5ef7-66b7-41db-ad79-3e917389a96f-istiod-ca-cert" pod "083c5ef7-66b7-41db-ad79-3e917389a96f" (UID: "083c5ef7-66b7-41db-ad79-3e917389a96f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:02:57.778221 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.778192 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-envoy" pod "083c5ef7-66b7-41db-ad79-3e917389a96f" (UID: "083c5ef7-66b7-41db-ad79-3e917389a96f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:02:57.778314 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.778243 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-podinfo" pod "083c5ef7-66b7-41db-ad79-3e917389a96f" (UID: "083c5ef7-66b7-41db-ad79-3e917389a96f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 16:02:57.778314 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.778272 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-kube-api-access-chfbf" pod "083c5ef7-66b7-41db-ad79-3e917389a96f" (UID: "083c5ef7-66b7-41db-ad79-3e917389a96f"). InnerVolumeSpecName "kube-api-access-chfbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:02:57.778314 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.778289 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-token" pod "083c5ef7-66b7-41db-ad79-3e917389a96f" (UID: "083c5ef7-66b7-41db-ad79-3e917389a96f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:02:57.877413 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.877379 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:57.877413 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.877407 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:57.877413 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.877418 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:57.877649 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.877436 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:57.877649 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.877450 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:57.877649 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.877462 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chfbf\" (UniqueName: \"kubernetes.io/projected/083c5ef7-66b7-41db-ad79-3e917389a96f-kube-api-access-chfbf\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:57.877649 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.877471 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083c5ef7-66b7-41db-ad79-3e917389a96f-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:57.877649 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:57.877479 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083c5ef7-66b7-41db-ad79-3e917389a96f-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:58.185800 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.185766 5433 generic.go:356] "Generic (PLEG): container finished" podID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerID="c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465" exitCode=0 Nov 02 16:02:58.186321 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.185811 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" Nov 02 16:02:58.186321 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.185847 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" event={"ID":"083c5ef7-66b7-41db-ad79-3e917389a96f","Type":"ContainerDied","Data":"c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465"} Nov 02 16:02:58.186321 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.185907 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg" event={"ID":"083c5ef7-66b7-41db-ad79-3e917389a96f","Type":"ContainerDied","Data":"fd90e388a20aaa740987a01a2ecbae84766793abc3397154c9415ed280c71e06"} Nov 02 16:02:58.186321 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.185930 5433 scope.go:122] "RemoveContainer" containerID="c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465" Nov 02 16:02:58.195287 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.195266 5433 scope.go:122] "RemoveContainer" containerID="496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180" Nov 02 16:02:58.203540 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.203520 5433 scope.go:122] "RemoveContainer" containerID="c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465" Nov 02 16:02:58.203789 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:02:58.203767 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465\": container with ID starting with c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465 not found: ID does not exist" containerID="c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465" Nov 02 16:02:58.203829 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.203799 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465"} err="failed to get container status \"c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465\": rpc error: code = NotFound desc = could not find container \"c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465\": container with ID starting with c34dedbc99986706e21632b3d471281cadd2d1f6150e8b395f67ba189f70a465 not found: ID does not exist" Nov 02 16:02:58.203829 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.203817 5433 scope.go:122] "RemoveContainer" containerID="496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180" Nov 02 16:02:58.204088 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:02:58.204070 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180\": container with ID starting with 496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180 not found: ID does not exist" containerID="496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180" Nov 02 16:02:58.204134 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.204096 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180"} err="failed to get container status \"496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180\": rpc error: code = NotFound desc = could not find container \"496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180\": container with ID starting with 496d1e76aa292283146b1f7779cb121454cd42abc4f07e9a29b84f746d410180 not found: ID does not exist" Nov 02 16:02:58.343892 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.343856 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg"] Nov 02 16:02:58.348095 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.348064 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1d7f44-openshift-default-77cd6959d9lkhrg"] Nov 02 16:02:58.455255 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:58.455185 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" path="/var/lib/kubelet/pods/083c5ef7-66b7-41db-ad79-3e917389a96f/volumes" Nov 02 16:02:59.641908 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:02:59.641868 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" podUID="69c42886-45ec-4ff8-849e-124210ee7b01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 16:03:05.424793 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:05.424769 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:03:05.548155 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:05.548071 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c42886-45ec-4ff8-849e-124210ee7b01-kube-api-access-68wh4\" (UniqueName: \"kubernetes.io/projected/69c42886-45ec-4ff8-849e-124210ee7b01-kube-api-access-68wh4\") pod \"69c42886-45ec-4ff8-849e-124210ee7b01\" (UID: \"69c42886-45ec-4ff8-849e-124210ee7b01\") " Nov 02 16:03:05.548155 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:05.548119 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c42886-45ec-4ff8-849e-124210ee7b01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c42886-45ec-4ff8-849e-124210ee7b01-plugins-config-volume\") pod \"69c42886-45ec-4ff8-849e-124210ee7b01\" (UID: \"69c42886-45ec-4ff8-849e-124210ee7b01\") " Nov 02 16:03:05.548526 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:05.548499 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c42886-45ec-4ff8-849e-124210ee7b01-plugins-config-volume" pod "69c42886-45ec-4ff8-849e-124210ee7b01" (UID: "69c42886-45ec-4ff8-849e-124210ee7b01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:03:05.550349 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:05.550324 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c42886-45ec-4ff8-849e-124210ee7b01-kube-api-access-68wh4" pod "69c42886-45ec-4ff8-849e-124210ee7b01" (UID: "69c42886-45ec-4ff8-849e-124210ee7b01"). InnerVolumeSpecName "kube-api-access-68wh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:03:05.649359 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:05.649322 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68wh4\" (UniqueName: \"kubernetes.io/projected/69c42886-45ec-4ff8-849e-124210ee7b01-kube-api-access-68wh4\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:03:05.649522 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:05.649363 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c42886-45ec-4ff8-849e-124210ee7b01-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:03:06.223595 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.223551 5433 generic.go:356] "Generic (PLEG): container finished" podID="69c42886-45ec-4ff8-849e-124210ee7b01" containerID="59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243" exitCode=0 Nov 02 16:03:06.223784 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.223602 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" Nov 02 16:03:06.223784 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.223628 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" event={"ID":"69c42886-45ec-4ff8-849e-124210ee7b01","Type":"ContainerDied","Data":"59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243"} Nov 02 16:03:06.223784 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.223684 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7" event={"ID":"69c42886-45ec-4ff8-849e-124210ee7b01","Type":"ContainerDied","Data":"44232c32fcb5d4c2c7e8e9acde86d032c60b63e908c6e225185345570a6e9f31"} Nov 02 16:03:06.223784 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.223708 5433 scope.go:122] "RemoveContainer" containerID="59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243" Nov 02 16:03:06.232364 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.232344 5433 scope.go:122] "RemoveContainer" containerID="59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243" Nov 02 16:03:06.232631 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:03:06.232611 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243\": container with ID starting with 59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243 not found: ID does not exist" containerID="59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243" Nov 02 16:03:06.232698 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.232644 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243"} err="failed to get container status \"59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243\": rpc error: code = NotFound desc = could not find container \"59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243\": container with ID starting with 59ecc46f98b4f3c3926c7d420a9432cc36672f32d3eb57bafe75542772da5243 not found: ID does not exist" Nov 02 16:03:06.377806 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.377766 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7"] Nov 02 16:03:06.424495 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.424461 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1d7f44-d77788455kslm7"] Nov 02 16:03:06.455820 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:06.455787 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c42886-45ec-4ff8-849e-124210ee7b01" path="/var/lib/kubelet/pods/69c42886-45ec-4ff8-849e-124210ee7b01/volumes" Nov 02 16:03:11.010103 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:03:11.010067 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:03:12.370171 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:12.370129 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:38.164495 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:38.164461 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:38.311859 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:38.311808 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:03:41.010391 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:03:41.010345 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:03:43.242197 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:03:43.242163 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:05.076424 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:04:05.076377 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:05.214825 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:04:05.214795 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:05.357800 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:04:05.357697 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:11.010943 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:04:11.010903 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:04:25.346932 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:04:25.346889 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:28.607036 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:04:28.606931 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:31.688196 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:04:31.688154 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:38.500047 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:04:38.500012 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:04:41.208491 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:04:41.208452 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:04:58.213647 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:04:58.213608 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:11.009953 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:05:11.009918 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:05:15.954197 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:15.954164 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b"] Nov 02 16:05:15.954648 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:15.954634 5433 cpu_manager.go:414] "containerMap: removing container" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" Nov 02 16:05:15.954695 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:15.954649 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" Nov 02 16:05:15.954738 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:15.954704 5433 cpu_manager.go:414] "containerMap: removing container" podUID="69c42886-45ec-4ff8-849e-124210ee7b01" containerName="payload-processing" Nov 02 16:05:15.954738 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:15.954711 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c42886-45ec-4ff8-849e-124210ee7b01" containerName="payload-processing" Nov 02 16:05:15.954811 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:15.954773 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" Nov 02 16:05:15.954811 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:15.954788 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" Nov 02 16:05:15.954811 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:15.954796 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c42886-45ec-4ff8-849e-124210ee7b01" containerName="payload-processing" Nov 02 16:05:15.958159 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:15.958138 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.023609 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.023575 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c97324a-1055-44db-85a9-568973dec114-maas-api-tls\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.023798 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.023637 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c97324a-1055-44db-85a9-568973dec114-metrics-certs\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.023798 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.023732 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c97324a-1055-44db-85a9-568973dec114-openshift-service-ca\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.023798 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.023785 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmxz\" (UniqueName: \"kubernetes.io/projected/4c97324a-1055-44db-85a9-568973dec114-kube-api-access-6kmxz\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.055680 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.055649 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b"] Nov 02 16:05:16.124780 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.124743 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c97324a-1055-44db-85a9-568973dec114-metrics-certs\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.125000 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.124796 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c97324a-1055-44db-85a9-568973dec114-openshift-service-ca\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.125000 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.124829 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmxz\" (UniqueName: \"kubernetes.io/projected/4c97324a-1055-44db-85a9-568973dec114-kube-api-access-6kmxz\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.125000 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.124884 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c97324a-1055-44db-85a9-568973dec114-maas-api-tls\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.125559 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.125537 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c97324a-1055-44db-85a9-568973dec114-openshift-service-ca\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.127291 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.127263 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c97324a-1055-44db-85a9-568973dec114-maas-api-tls\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.127291 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.127275 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c97324a-1055-44db-85a9-568973dec114-metrics-certs\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.182786 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.182752 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmxz\" (UniqueName: \"kubernetes.io/projected/4c97324a-1055-44db-85a9-568973dec114-kube-api-access-6kmxz\") pod \"maas-api-57678c84d7-mnf4b\" (UID: \"4c97324a-1055-44db-85a9-568973dec114\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.272204 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.272115 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:16.641006 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.640838 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b"] Nov 02 16:05:16.644109 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:05:16.644080 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c97324a_1055_44db_85a9_568973dec114.slice/crio-56d6748a0d536e209fee7fffdad22255bac15e24e98ca86bc65fab32e1199d35 WatchSource:0}: Error finding container 56d6748a0d536e209fee7fffdad22255bac15e24e98ca86bc65fab32e1199d35: Status 404 returned error can't find the container with id 56d6748a0d536e209fee7fffdad22255bac15e24e98ca86bc65fab32e1199d35 Nov 02 16:05:16.645729 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:16.645712 5433 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 16:05:17.062072 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:17.061983 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" event={"ID":"4c97324a-1055-44db-85a9-568973dec114","Type":"ContainerStarted","Data":"56d6748a0d536e209fee7fffdad22255bac15e24e98ca86bc65fab32e1199d35"} Nov 02 16:05:17.850110 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:17.850024 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:18.068400 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:18.068360 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" event={"ID":"4c97324a-1055-44db-85a9-568973dec114","Type":"ContainerStarted","Data":"9fe311fadf54968f0bb4267163d9e4890b36e81e506b11ea72b246101dfe09c4"} Nov 02 16:05:18.068872 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:18.068427 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:18.181913 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:18.181859 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" podStartSLOduration=2.25021964 podStartE2EDuration="3.181843538s" totalImagesPullingTime="931.623898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:05:18.180225281 +0000 UTC m=+4021.853428471" watchObservedRunningTime="2026-11-02 16:05:18.181843538 +0000 UTC m=+4021.855046733" Nov 02 16:05:22.275607 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:22.275570 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:22.885436 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:22.885382 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:23.053204 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:23.053176 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:24.085571 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:24.085537 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57678c84d7-mnf4b" Nov 02 16:05:25.968456 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:25.968411 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:26.136260 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:26.136225 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:26.287457 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:26.287372 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:29.129406 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.129373 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:29.307265 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.307234 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:29.542451 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.542373 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl"] Nov 02 16:05:29.542808 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.542793 5433 cpu_manager.go:414] "containerMap: removing container" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" Nov 02 16:05:29.542858 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.542811 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083c5ef7-66b7-41db-ad79-3e917389a96f" containerName="istio-proxy" Nov 02 16:05:29.545763 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.545745 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:29.573951 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.573921 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2t7l\" (UniqueName: \"kubernetes.io/projected/009477c5-3b48-40ec-a800-6571dd7ffa8f-kube-api-access-z2t7l\") pod \"payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl\" (UID: \"009477c5-3b48-40ec-a800-6571dd7ffa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:29.574103 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.574086 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009477c5-3b48-40ec-a800-6571dd7ffa8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl\" (UID: \"009477c5-3b48-40ec-a800-6571dd7ffa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:29.644005 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.641615 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f22ac84e\"" Nov 02 16:05:29.644005 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.641709 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f22ac84e-dockercfg-kh2vr\"" Nov 02 16:05:29.645969 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.645940 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl"] Nov 02 16:05:29.674935 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.674904 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2t7l\" (UniqueName: \"kubernetes.io/projected/009477c5-3b48-40ec-a800-6571dd7ffa8f-kube-api-access-z2t7l\") pod \"payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl\" (UID: \"009477c5-3b48-40ec-a800-6571dd7ffa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:29.675146 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.675025 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009477c5-3b48-40ec-a800-6571dd7ffa8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl\" (UID: \"009477c5-3b48-40ec-a800-6571dd7ffa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:29.675732 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.675715 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009477c5-3b48-40ec-a800-6571dd7ffa8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl\" (UID: \"009477c5-3b48-40ec-a800-6571dd7ffa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:29.699267 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.699243 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2"] Nov 02 16:05:29.702812 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.702796 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:29.733149 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.733120 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2t7l\" (UniqueName: \"kubernetes.io/projected/009477c5-3b48-40ec-a800-6571dd7ffa8f-kube-api-access-z2t7l\") pod \"payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl\" (UID: \"009477c5-3b48-40ec-a800-6571dd7ffa8f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:29.758599 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.758571 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2"] Nov 02 16:05:29.776080 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.776047 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7570733-e3d7-4831-8938-9d559463896d-plugins-config-volume\") pod \"payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2\" (UID: \"c7570733-e3d7-4831-8938-9d559463896d\") " pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:29.776214 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.776156 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bnkc\" (UniqueName: \"kubernetes.io/projected/c7570733-e3d7-4831-8938-9d559463896d-kube-api-access-4bnkc\") pod \"payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2\" (UID: \"c7570733-e3d7-4831-8938-9d559463896d\") " pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:29.858506 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.858422 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:29.877398 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.877366 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bnkc\" (UniqueName: \"kubernetes.io/projected/c7570733-e3d7-4831-8938-9d559463896d-kube-api-access-4bnkc\") pod \"payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2\" (UID: \"c7570733-e3d7-4831-8938-9d559463896d\") " pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:29.877532 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.877499 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7570733-e3d7-4831-8938-9d559463896d-plugins-config-volume\") pod \"payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2\" (UID: \"c7570733-e3d7-4831-8938-9d559463896d\") " pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:29.878204 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.878181 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7570733-e3d7-4831-8938-9d559463896d-plugins-config-volume\") pod \"payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2\" (UID: \"c7570733-e3d7-4831-8938-9d559463896d\") " pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:29.935448 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:29.935415 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bnkc\" (UniqueName: \"kubernetes.io/projected/c7570733-e3d7-4831-8938-9d559463896d-kube-api-access-4bnkc\") pod \"payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2\" (UID: \"c7570733-e3d7-4831-8938-9d559463896d\") " pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:30.015605 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.015573 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:30.222659 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.222627 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl"] Nov 02 16:05:30.225449 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:05:30.225411 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009477c5_3b48_40ec_a800_6571dd7ffa8f.slice/crio-6f345aa0cdc751a118fa651d3918253436dedc5a5395951df248014e58f9a8f1 WatchSource:0}: Error finding container 6f345aa0cdc751a118fa651d3918253436dedc5a5395951df248014e58f9a8f1: Status 404 returned error can't find the container with id 6f345aa0cdc751a118fa651d3918253436dedc5a5395951df248014e58f9a8f1 Nov 02 16:05:30.382678 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.382651 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2"] Nov 02 16:05:30.386832 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:05:30.386799 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7570733_e3d7_4831_8938_9d559463896d.slice/crio-c990947f3c7550a0c6fdf881ecbd635598ec025573826bdd33103d930eec2540 WatchSource:0}: Error finding container c990947f3c7550a0c6fdf881ecbd635598ec025573826bdd33103d930eec2540: Status 404 returned error can't find the container with id c990947f3c7550a0c6fdf881ecbd635598ec025573826bdd33103d930eec2540 Nov 02 16:05:30.714350 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.714313 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r"] Nov 02 16:05:30.722887 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.719208 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.769123 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.769088 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f22ac84e-serving-cert\"" Nov 02 16:05:30.774817 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.774719 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r"] Nov 02 16:05:30.786479 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.786447 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4428\" (UniqueName: \"kubernetes.io/projected/17db4e16-ece1-4149-86ef-f664dc24ad16-kube-api-access-l4428\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.786635 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.786513 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.786635 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.786599 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-metrics-certs\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.786635 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.786629 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db4e16-ece1-4149-86ef-f664dc24ad16-openshift-service-ca\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.887446 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.887409 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4428\" (UniqueName: \"kubernetes.io/projected/17db4e16-ece1-4149-86ef-f664dc24ad16-kube-api-access-l4428\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.887640 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.887467 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.887640 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.887493 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-metrics-certs\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.887640 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.887509 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db4e16-ece1-4149-86ef-f664dc24ad16-openshift-service-ca\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.887640 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:05:30.887626 5433 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-serving-cert: secret "maas-api-e2e-ait-f22ac84e-serving-cert" not found Nov 02 16:05:30.887854 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:05:30.887686 5433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls podName:17db4e16-ece1-4149-86ef-f664dc24ad16 nodeName:}" failed. No retries permitted until 2026-11-02 16:05:31.387669445 +0000 UTC m=+4035.060872635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls") pod "maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" (UID: "17db4e16-ece1-4149-86ef-f664dc24ad16") : secret "maas-api-e2e-ait-f22ac84e-serving-cert" not found Nov 02 16:05:30.888154 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.888137 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db4e16-ece1-4149-86ef-f664dc24ad16-openshift-service-ca\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.890464 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.890419 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-metrics-certs\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.947598 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.947561 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4428\" (UniqueName: \"kubernetes.io/projected/17db4e16-ece1-4149-86ef-f664dc24ad16-kube-api-access-l4428\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:30.966896 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:30.966816 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:31.113923 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.113884 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:31.131635 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.131598 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" event={"ID":"c7570733-e3d7-4831-8938-9d559463896d","Type":"ContainerStarted","Data":"9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e"} Nov 02 16:05:31.131812 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.131664 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" event={"ID":"c7570733-e3d7-4831-8938-9d559463896d","Type":"ContainerStarted","Data":"c990947f3c7550a0c6fdf881ecbd635598ec025573826bdd33103d930eec2540"} Nov 02 16:05:31.131812 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.131711 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:31.133619 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.133585 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" event={"ID":"009477c5-3b48-40ec-a800-6571dd7ffa8f","Type":"ContainerStarted","Data":"b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348"} Nov 02 16:05:31.133773 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.133631 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" event={"ID":"009477c5-3b48-40ec-a800-6571dd7ffa8f","Type":"ContainerStarted","Data":"6f345aa0cdc751a118fa651d3918253436dedc5a5395951df248014e58f9a8f1"} Nov 02 16:05:31.133773 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.133657 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:31.242778 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.242657 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" podStartSLOduration=2.242639941 podStartE2EDuration="2.242639941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:05:31.241735332 +0000 UTC m=+4034.914938551" watchObservedRunningTime="2026-11-02 16:05:31.242639941 +0000 UTC m=+4034.915843137" Nov 02 16:05:31.352216 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.352137 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" podStartSLOduration=2.352115921 podStartE2EDuration="2.352115921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:05:31.350472031 +0000 UTC m=+4035.023675227" watchObservedRunningTime="2026-11-02 16:05:31.352115921 +0000 UTC m=+4035.025319117" Nov 02 16:05:31.367062 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.366961 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:31.392535 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.392497 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:31.395010 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.394987 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls\") pod \"maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:31.635989 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:31.635952 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:32.050142 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:32.050117 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r"] Nov 02 16:05:32.052918 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:05:32.052891 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17db4e16_ece1_4149_86ef_f664dc24ad16.slice/crio-4f1e35c2b9aeac13a11d63c14be6ba283db960baa1ecb71d37f13f8e3f836f34 WatchSource:0}: Error finding container 4f1e35c2b9aeac13a11d63c14be6ba283db960baa1ecb71d37f13f8e3f836f34: Status 404 returned error can't find the container with id 4f1e35c2b9aeac13a11d63c14be6ba283db960baa1ecb71d37f13f8e3f836f34 Nov 02 16:05:32.139629 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:32.139589 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" event={"ID":"17db4e16-ece1-4149-86ef-f664dc24ad16","Type":"ContainerStarted","Data":"4f1e35c2b9aeac13a11d63c14be6ba283db960baa1ecb71d37f13f8e3f836f34"} Nov 02 16:05:33.146297 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:33.146240 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" event={"ID":"17db4e16-ece1-4149-86ef-f664dc24ad16","Type":"ContainerStarted","Data":"22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595"} Nov 02 16:05:33.146833 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:33.146470 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:33.258838 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:33.258713 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" podStartSLOduration=2.375260052 podStartE2EDuration="3.258695337s" totalImagesPullingTime="883.435285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:05:33.257676369 +0000 UTC m=+4036.930879590" watchObservedRunningTime="2026-11-02 16:05:33.258695337 +0000 UTC m=+4036.931898532" Nov 02 16:05:33.584463 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:33.584380 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:33.736144 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:33.736111 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:33.941270 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:33.941239 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-nj427"] Nov 02 16:05:33.941460 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:33.941435 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" podUID="010ec615-c5e2-4791-94a8-3951ce25ec1a" containerName="maas-api" containerID="cri-o://ec18a182f228ac04552badcea50a92962fa9f561f9d7f58158487610485d7256" gracePeriod=30 Nov 02 16:05:34.524520 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:34.524475 5433 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" podUID="010ec615-c5e2-4791-94a8-3951ce25ec1a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.28:8443/health\": dial tcp 10.133.0.28:8443: connect: connection refused" Nov 02 16:05:35.158313 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.158283 5433 generic.go:356] "Generic (PLEG): container finished" podID="010ec615-c5e2-4791-94a8-3951ce25ec1a" containerID="ec18a182f228ac04552badcea50a92962fa9f561f9d7f58158487610485d7256" exitCode=0 Nov 02 16:05:35.158466 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.158335 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" event={"ID":"010ec615-c5e2-4791-94a8-3951ce25ec1a","Type":"ContainerDied","Data":"ec18a182f228ac04552badcea50a92962fa9f561f9d7f58158487610485d7256"} Nov 02 16:05:35.191371 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.191342 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 16:05:35.229805 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.229783 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-maas-api-tls\") pod \"010ec615-c5e2-4791-94a8-3951ce25ec1a\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " Nov 02 16:05:35.229923 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.229814 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-metrics-certs\") pod \"010ec615-c5e2-4791-94a8-3951ce25ec1a\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " Nov 02 16:05:35.229923 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.229843 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010ec615-c5e2-4791-94a8-3951ce25ec1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010ec615-c5e2-4791-94a8-3951ce25ec1a-openshift-service-ca\") pod \"010ec615-c5e2-4791-94a8-3951ce25ec1a\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " Nov 02 16:05:35.229923 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.229867 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010ec615-c5e2-4791-94a8-3951ce25ec1a-kube-api-access-wm5jp\" (UniqueName: \"kubernetes.io/projected/010ec615-c5e2-4791-94a8-3951ce25ec1a-kube-api-access-wm5jp\") pod \"010ec615-c5e2-4791-94a8-3951ce25ec1a\" (UID: \"010ec615-c5e2-4791-94a8-3951ce25ec1a\") " Nov 02 16:05:35.230231 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.230197 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010ec615-c5e2-4791-94a8-3951ce25ec1a-openshift-service-ca" pod "010ec615-c5e2-4791-94a8-3951ce25ec1a" (UID: "010ec615-c5e2-4791-94a8-3951ce25ec1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:05:35.231809 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.231783 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-maas-api-tls" pod "010ec615-c5e2-4791-94a8-3951ce25ec1a" (UID: "010ec615-c5e2-4791-94a8-3951ce25ec1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:05:35.231963 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.231942 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010ec615-c5e2-4791-94a8-3951ce25ec1a-kube-api-access-wm5jp" pod "010ec615-c5e2-4791-94a8-3951ce25ec1a" (UID: "010ec615-c5e2-4791-94a8-3951ce25ec1a"). InnerVolumeSpecName "kube-api-access-wm5jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:05:35.232275 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.232251 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-metrics-certs" pod "010ec615-c5e2-4791-94a8-3951ce25ec1a" (UID: "010ec615-c5e2-4791-94a8-3951ce25ec1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:05:35.331687 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.331616 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:35.331687 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.331653 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010ec615-c5e2-4791-94a8-3951ce25ec1a-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:35.331687 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.331671 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010ec615-c5e2-4791-94a8-3951ce25ec1a-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:35.331687 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.331686 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm5jp\" (UniqueName: \"kubernetes.io/projected/010ec615-c5e2-4791-94a8-3951ce25ec1a-kube-api-access-wm5jp\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:35.940922 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:35.940885 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:36.164644 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:36.164609 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" event={"ID":"010ec615-c5e2-4791-94a8-3951ce25ec1a","Type":"ContainerDied","Data":"c5ea29abade11f85003fafc87b6354d9b5c491aba9601421c302ff16ca1e0b55"} Nov 02 16:05:36.164644 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:36.164637 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-nj427" Nov 02 16:05:36.164872 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:36.164702 5433 scope.go:122] "RemoveContainer" containerID="ec18a182f228ac04552badcea50a92962fa9f561f9d7f58158487610485d7256" Nov 02 16:05:36.319296 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:36.319213 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-nj427"] Nov 02 16:05:36.366414 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:36.366382 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-nj427"] Nov 02 16:05:36.455892 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:36.455856 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010ec615-c5e2-4791-94a8-3951ce25ec1a" path="/var/lib/kubelet/pods/010ec615-c5e2-4791-94a8-3951ce25ec1a/volumes" Nov 02 16:05:38.815669 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:38.815627 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:39.157118 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:39.157089 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:05:40.951851 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:40.951819 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:41.027994 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:05:41.027950 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:05:42.142725 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:42.142694 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:05:42.143243 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:42.142762 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:05:45.635413 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:45.635374 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:50.621152 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:50.621118 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:05:55.621194 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:05:55.621151 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:00.412489 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:00.412395 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:05.110087 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:05.110038 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:09.831224 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:09.831187 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:11.011168 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:06:11.011132 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:06:13.826508 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:13.826472 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2"] Nov 02 16:06:13.927126 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:13.927094 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl"] Nov 02 16:06:14.178600 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:14.178565 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:14.328719 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:14.328687 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:14.482739 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:14.482608 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:18.422990 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.422936 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r"] Nov 02 16:06:18.423409 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.423151 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" podUID="17db4e16-ece1-4149-86ef-f664dc24ad16" containerName="maas-api" containerID="cri-o://22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595" gracePeriod=30 Nov 02 16:06:18.541839 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.541807 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:18.697536 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.697509 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:06:18.801606 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.801561 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17db4e16-ece1-4149-86ef-f664dc24ad16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db4e16-ece1-4149-86ef-f664dc24ad16-openshift-service-ca\") pod \"17db4e16-ece1-4149-86ef-f664dc24ad16\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " Nov 02 16:06:18.801829 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.801665 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-metrics-certs\") pod \"17db4e16-ece1-4149-86ef-f664dc24ad16\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " Nov 02 16:06:18.801829 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.801709 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17db4e16-ece1-4149-86ef-f664dc24ad16-kube-api-access-l4428\" (UniqueName: \"kubernetes.io/projected/17db4e16-ece1-4149-86ef-f664dc24ad16-kube-api-access-l4428\") pod \"17db4e16-ece1-4149-86ef-f664dc24ad16\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " Nov 02 16:06:18.801829 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.801761 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls\") pod \"17db4e16-ece1-4149-86ef-f664dc24ad16\" (UID: \"17db4e16-ece1-4149-86ef-f664dc24ad16\") " Nov 02 16:06:18.802154 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.802087 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17db4e16-ece1-4149-86ef-f664dc24ad16-openshift-service-ca" pod "17db4e16-ece1-4149-86ef-f664dc24ad16" (UID: "17db4e16-ece1-4149-86ef-f664dc24ad16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:06:18.804907 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.804757 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls" pod "17db4e16-ece1-4149-86ef-f664dc24ad16" (UID: "17db4e16-ece1-4149-86ef-f664dc24ad16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:06:18.804907 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.804867 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17db4e16-ece1-4149-86ef-f664dc24ad16-kube-api-access-l4428" pod "17db4e16-ece1-4149-86ef-f664dc24ad16" (UID: "17db4e16-ece1-4149-86ef-f664dc24ad16"). InnerVolumeSpecName "kube-api-access-l4428". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:06:18.805397 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.805373 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-metrics-certs" pod "17db4e16-ece1-4149-86ef-f664dc24ad16" (UID: "17db4e16-ece1-4149-86ef-f664dc24ad16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:06:18.827162 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.827085 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" podUID="c7570733-e3d7-4831-8938-9d559463896d" containerName="payload-processing" containerID="cri-o://9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e" gracePeriod=25 Nov 02 16:06:18.903204 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.903174 5433 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17db4e16-ece1-4149-86ef-f664dc24ad16-openshift-service-ca\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:18.903204 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.903202 5433 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-metrics-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:18.903396 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.903213 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4428\" (UniqueName: \"kubernetes.io/projected/17db4e16-ece1-4149-86ef-f664dc24ad16-kube-api-access-l4428\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:18.903396 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.903222 5433 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17db4e16-ece1-4149-86ef-f664dc24ad16-maas-api-tls\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:18.931193 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:18.931060 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" podUID="009477c5-3b48-40ec-a800-6571dd7ffa8f" containerName="payload-pre-processing" containerID="cri-o://b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348" gracePeriod=25 Nov 02 16:06:19.071292 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.071268 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:06:19.181375 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.181310 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:06:19.206048 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.206001 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7570733-e3d7-4831-8938-9d559463896d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7570733-e3d7-4831-8938-9d559463896d-plugins-config-volume\") pod \"c7570733-e3d7-4831-8938-9d559463896d\" (UID: \"c7570733-e3d7-4831-8938-9d559463896d\") " Nov 02 16:06:19.206236 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.206062 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7570733-e3d7-4831-8938-9d559463896d-kube-api-access-4bnkc\" (UniqueName: \"kubernetes.io/projected/c7570733-e3d7-4831-8938-9d559463896d-kube-api-access-4bnkc\") pod \"c7570733-e3d7-4831-8938-9d559463896d\" (UID: \"c7570733-e3d7-4831-8938-9d559463896d\") " Nov 02 16:06:19.206475 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.206443 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7570733-e3d7-4831-8938-9d559463896d-plugins-config-volume" pod "c7570733-e3d7-4831-8938-9d559463896d" (UID: "c7570733-e3d7-4831-8938-9d559463896d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:06:19.208763 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.208734 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7570733-e3d7-4831-8938-9d559463896d-kube-api-access-4bnkc" pod "c7570733-e3d7-4831-8938-9d559463896d" (UID: "c7570733-e3d7-4831-8938-9d559463896d"). InnerVolumeSpecName "kube-api-access-4bnkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:06:19.307137 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.307101 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009477c5-3b48-40ec-a800-6571dd7ffa8f-kube-api-access-z2t7l\" (UniqueName: \"kubernetes.io/projected/009477c5-3b48-40ec-a800-6571dd7ffa8f-kube-api-access-z2t7l\") pod \"009477c5-3b48-40ec-a800-6571dd7ffa8f\" (UID: \"009477c5-3b48-40ec-a800-6571dd7ffa8f\") " Nov 02 16:06:19.307137 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.307140 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009477c5-3b48-40ec-a800-6571dd7ffa8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009477c5-3b48-40ec-a800-6571dd7ffa8f-plugins-config-volume\") pod \"009477c5-3b48-40ec-a800-6571dd7ffa8f\" (UID: \"009477c5-3b48-40ec-a800-6571dd7ffa8f\") " Nov 02 16:06:19.307529 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.307507 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7570733-e3d7-4831-8938-9d559463896d-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:19.307625 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.307536 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bnkc\" (UniqueName: \"kubernetes.io/projected/c7570733-e3d7-4831-8938-9d559463896d-kube-api-access-4bnkc\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:19.307625 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.307583 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009477c5-3b48-40ec-a800-6571dd7ffa8f-plugins-config-volume" pod "009477c5-3b48-40ec-a800-6571dd7ffa8f" (UID: "009477c5-3b48-40ec-a800-6571dd7ffa8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:06:19.309412 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.309387 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009477c5-3b48-40ec-a800-6571dd7ffa8f-kube-api-access-z2t7l" pod "009477c5-3b48-40ec-a800-6571dd7ffa8f" (UID: "009477c5-3b48-40ec-a800-6571dd7ffa8f"). InnerVolumeSpecName "kube-api-access-z2t7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:06:19.386215 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.386181 5433 generic.go:356] "Generic (PLEG): container finished" podID="c7570733-e3d7-4831-8938-9d559463896d" containerID="9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e" exitCode=0 Nov 02 16:06:19.386417 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.386243 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" Nov 02 16:06:19.386417 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.386265 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" event={"ID":"c7570733-e3d7-4831-8938-9d559463896d","Type":"ContainerDied","Data":"9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e"} Nov 02 16:06:19.386417 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.386358 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2" event={"ID":"c7570733-e3d7-4831-8938-9d559463896d","Type":"ContainerDied","Data":"c990947f3c7550a0c6fdf881ecbd635598ec025573826bdd33103d930eec2540"} Nov 02 16:06:19.386417 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.386389 5433 scope.go:122] "RemoveContainer" containerID="9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e" Nov 02 16:06:19.387721 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.387698 5433 generic.go:356] "Generic (PLEG): container finished" podID="009477c5-3b48-40ec-a800-6571dd7ffa8f" containerID="b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348" exitCode=0 Nov 02 16:06:19.387843 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.387728 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" event={"ID":"009477c5-3b48-40ec-a800-6571dd7ffa8f","Type":"ContainerDied","Data":"b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348"} Nov 02 16:06:19.387843 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.387782 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" event={"ID":"009477c5-3b48-40ec-a800-6571dd7ffa8f","Type":"ContainerDied","Data":"6f345aa0cdc751a118fa651d3918253436dedc5a5395951df248014e58f9a8f1"} Nov 02 16:06:19.388101 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.388078 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl" Nov 02 16:06:19.389222 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.389201 5433 generic.go:356] "Generic (PLEG): container finished" podID="17db4e16-ece1-4149-86ef-f664dc24ad16" containerID="22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595" exitCode=0 Nov 02 16:06:19.389340 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.389229 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" event={"ID":"17db4e16-ece1-4149-86ef-f664dc24ad16","Type":"ContainerDied","Data":"22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595"} Nov 02 16:06:19.389340 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.389264 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" Nov 02 16:06:19.389340 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.389279 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r" event={"ID":"17db4e16-ece1-4149-86ef-f664dc24ad16","Type":"ContainerDied","Data":"4f1e35c2b9aeac13a11d63c14be6ba283db960baa1ecb71d37f13f8e3f836f34"} Nov 02 16:06:19.396185 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.396166 5433 scope.go:122] "RemoveContainer" containerID="9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e" Nov 02 16:06:19.396443 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:06:19.396424 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e\": container with ID starting with 9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e not found: ID does not exist" containerID="9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e" Nov 02 16:06:19.396534 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.396452 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e"} err="failed to get container status \"9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e\": rpc error: code = NotFound desc = could not find container \"9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e\": container with ID starting with 9442ce1e3e530ea3644da2f8e2e10a26ce819b35c234e448a07dc7994f433e3e not found: ID does not exist" Nov 02 16:06:19.396534 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.396469 5433 scope.go:122] "RemoveContainer" containerID="b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348" Nov 02 16:06:19.405773 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.405757 5433 scope.go:122] "RemoveContainer" containerID="b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348" Nov 02 16:06:19.406108 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:06:19.406086 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348\": container with ID starting with b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348 not found: ID does not exist" containerID="b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348" Nov 02 16:06:19.406163 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.406118 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348"} err="failed to get container status \"b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348\": rpc error: code = NotFound desc = could not find container \"b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348\": container with ID starting with b60bcbcb78142a039f17ae22f98cbb95047592fa225c77916f6edfbac3f8b348 not found: ID does not exist" Nov 02 16:06:19.406163 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.406145 5433 scope.go:122] "RemoveContainer" containerID="22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595" Nov 02 16:06:19.408352 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.408332 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2t7l\" (UniqueName: \"kubernetes.io/projected/009477c5-3b48-40ec-a800-6571dd7ffa8f-kube-api-access-z2t7l\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:19.408432 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.408356 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009477c5-3b48-40ec-a800-6571dd7ffa8f-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:19.413891 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.413873 5433 scope.go:122] "RemoveContainer" containerID="22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595" Nov 02 16:06:19.414240 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:06:19.414218 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595\": container with ID starting with 22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595 not found: ID does not exist" containerID="22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595" Nov 02 16:06:19.414308 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.414250 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595"} err="failed to get container status \"22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595\": rpc error: code = NotFound desc = could not find container \"22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595\": container with ID starting with 22448a27607ab4fdb56627fb9e051ebf08f03bd5afc0d952c213b19299341595 not found: ID does not exist" Nov 02 16:06:19.541685 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.541648 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl"] Nov 02 16:06:19.545631 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.545603 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f22ac84e-c6f9749f9-2fbvl"] Nov 02 16:06:19.648821 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.648782 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2"] Nov 02 16:06:19.652884 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.652856 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f22ac84e-6b486ff98f-vt4s2"] Nov 02 16:06:19.756425 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.756342 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r"] Nov 02 16:06:19.762647 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.762620 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f22ac84e-758f7576d4-7k47r"] Nov 02 16:06:19.993507 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:19.993473 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:20.336656 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:20.336622 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:20.455401 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:20.455367 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009477c5-3b48-40ec-a800-6571dd7ffa8f" path="/var/lib/kubelet/pods/009477c5-3b48-40ec-a800-6571dd7ffa8f/volumes" Nov 02 16:06:20.455790 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:20.455775 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17db4e16-ece1-4149-86ef-f664dc24ad16" path="/var/lib/kubelet/pods/17db4e16-ece1-4149-86ef-f664dc24ad16/volumes" Nov 02 16:06:20.456150 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:20.456136 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7570733-e3d7-4831-8938-9d559463896d" path="/var/lib/kubelet/pods/c7570733-e3d7-4831-8938-9d559463896d/volumes" Nov 02 16:06:23.466649 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:23.466620 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:23.625142 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:23.624467 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:23.784706 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:23.784594 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:24.725087 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:24.725045 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:25.314286 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314256 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2"] Nov 02 16:06:25.314710 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314695 5433 cpu_manager.go:414] "containerMap: removing container" podUID="c7570733-e3d7-4831-8938-9d559463896d" containerName="payload-processing" Nov 02 16:06:25.314762 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314711 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7570733-e3d7-4831-8938-9d559463896d" containerName="payload-processing" Nov 02 16:06:25.314762 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314728 5433 cpu_manager.go:414] "containerMap: removing container" podUID="17db4e16-ece1-4149-86ef-f664dc24ad16" containerName="maas-api" Nov 02 16:06:25.314762 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314733 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17db4e16-ece1-4149-86ef-f664dc24ad16" containerName="maas-api" Nov 02 16:06:25.314762 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314743 5433 cpu_manager.go:414] "containerMap: removing container" podUID="010ec615-c5e2-4791-94a8-3951ce25ec1a" containerName="maas-api" Nov 02 16:06:25.314762 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314748 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010ec615-c5e2-4791-94a8-3951ce25ec1a" containerName="maas-api" Nov 02 16:06:25.314762 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314758 5433 cpu_manager.go:414] "containerMap: removing container" podUID="009477c5-3b48-40ec-a800-6571dd7ffa8f" containerName="payload-pre-processing" Nov 02 16:06:25.314762 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314763 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009477c5-3b48-40ec-a800-6571dd7ffa8f" containerName="payload-pre-processing" Nov 02 16:06:25.314992 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314817 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="010ec615-c5e2-4791-94a8-3951ce25ec1a" containerName="maas-api" Nov 02 16:06:25.314992 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314827 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="009477c5-3b48-40ec-a800-6571dd7ffa8f" containerName="payload-pre-processing" Nov 02 16:06:25.314992 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314837 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7570733-e3d7-4831-8938-9d559463896d" containerName="payload-processing" Nov 02 16:06:25.314992 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.314846 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="17db4e16-ece1-4149-86ef-f664dc24ad16" containerName="maas-api" Nov 02 16:06:25.318044 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.318019 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.378617 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.378568 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-socket\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.378834 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.378629 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f374c6-0f05-4faf-9aee-99c9bb843595-istiod-ca-cert\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.378834 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.378674 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsv2c\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-kube-api-access-jsv2c\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.378834 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.378733 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-podinfo\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.378834 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.378761 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-certs\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.379192 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.378937 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-envoy\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.379192 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.378998 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-token\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.379192 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.379051 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-credential-socket\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.379192 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.379075 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-715ca8-openshift-default-dockercfg-2t7sc\"" Nov 02 16:06:25.379192 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.379095 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-data\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.426857 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.426821 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2"] Nov 02 16:06:25.436211 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.436178 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:25.480626 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.480590 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-socket\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.480794 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.480633 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f374c6-0f05-4faf-9aee-99c9bb843595-istiod-ca-cert\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.480794 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.480672 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsv2c\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-kube-api-access-jsv2c\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.480794 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.480713 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-podinfo\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.480794 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.480743 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-certs\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.480992 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.480799 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-envoy\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.480992 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.480832 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-token\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.480992 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.480880 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-credential-socket\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.480992 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.480915 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-data\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.481346 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.481319 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-socket\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.481475 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.481343 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-certs\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.481475 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.481395 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-data\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.481475 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.481463 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-credential-socket\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.481736 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.481713 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f374c6-0f05-4faf-9aee-99c9bb843595-istiod-ca-cert\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.483287 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.483266 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-envoy\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.483716 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.483693 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-podinfo\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.539025 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.538989 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-token\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.539197 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.539101 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsv2c\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-kube-api-access-jsv2c\") pod \"e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:25.608392 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.608299 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:25.631686 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:25.631654 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:26.051295 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:26.051203 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2"] Nov 02 16:06:26.053912 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:06:26.053882 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f374c6_0f05_4faf_9aee_99c9bb843595.slice/crio-ec60d83165334bfa5855d46cd51dbc632a9725d520bbf6f992098a9b9c9e749a WatchSource:0}: Error finding container ec60d83165334bfa5855d46cd51dbc632a9725d520bbf6f992098a9b9c9e749a: Status 404 returned error can't find the container with id ec60d83165334bfa5855d46cd51dbc632a9725d520bbf6f992098a9b9c9e749a Nov 02 16:06:26.056109 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:26.056073 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:06:26.056225 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:26.056140 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:06:26.056225 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:26.056167 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:06:26.435056 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:26.435020 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" event={"ID":"48f374c6-0f05-4faf-9aee-99c9bb843595","Type":"ContainerStarted","Data":"8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb"} Nov 02 16:06:26.435203 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:26.435081 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" event={"ID":"48f374c6-0f05-4faf-9aee-99c9bb843595","Type":"ContainerStarted","Data":"ec60d83165334bfa5855d46cd51dbc632a9725d520bbf6f992098a9b9c9e749a"} Nov 02 16:06:26.551425 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:26.551380 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" podStartSLOduration=1.55136677 podStartE2EDuration="1.55136677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:06:26.550388262 +0000 UTC m=+4090.223591460" watchObservedRunningTime="2026-11-02 16:06:26.55136677 +0000 UTC m=+4090.224569965" Nov 02 16:06:26.632796 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:26.632757 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:27.638354 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:27.638321 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:28.444341 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:28.444294 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:28.445467 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:28.445446 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:29.314197 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:29.314162 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:29.492667 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:29.492633 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:29.647248 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:29.647208 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:32.527406 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:32.527313 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:32.698814 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:32.698781 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:32.948029 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:32.947993 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf"] Nov 02 16:06:32.951749 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:32.951726 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:33.001432 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.001401 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-715ca8\"" Nov 02 16:06:33.048161 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.048130 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-715ca8-dockercfg-n8p2h\"" Nov 02 16:06:33.051529 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.051460 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf"] Nov 02 16:06:33.072583 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.072535 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m59lj\" (UniqueName: \"kubernetes.io/projected/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-kube-api-access-m59lj\") pod \"payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf\" (UID: \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:33.072799 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.072623 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf\" (UID: \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:33.100854 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.100822 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p"] Nov 02 16:06:33.108221 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.108189 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:33.161674 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.161639 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p"] Nov 02 16:06:33.173434 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.173408 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m59lj\" (UniqueName: \"kubernetes.io/projected/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-kube-api-access-m59lj\") pod \"payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf\" (UID: \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:33.173597 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.173454 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf\" (UID: \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:33.174155 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.174134 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf\" (UID: \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:33.231803 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.231742 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m59lj\" (UniqueName: \"kubernetes.io/projected/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-kube-api-access-m59lj\") pod \"payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf\" (UID: \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:33.265302 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.265271 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:33.274777 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.274750 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p\" (UID: \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:33.274888 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.274872 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwg5\" (UniqueName: \"kubernetes.io/projected/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-kube-api-access-pgwg5\") pod \"payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p\" (UID: \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:33.375742 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.375705 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p\" (UID: \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:33.375930 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.375821 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwg5\" (UniqueName: \"kubernetes.io/projected/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-kube-api-access-pgwg5\") pod \"payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p\" (UID: \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:33.376460 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.376439 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p\" (UID: \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:33.433944 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.433914 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwg5\" (UniqueName: \"kubernetes.io/projected/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-kube-api-access-pgwg5\") pod \"payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p\" (UID: \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:33.640274 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.640137 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf"] Nov 02 16:06:33.643384 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:06:33.643354 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4d7283_5eba_40e7_9c1e_e0a7b2d677c7.slice/crio-0753e8d2407b76c76797b21113ac30f2d85d9bfa9eb96ccd7a08d2847da7ebe0 WatchSource:0}: Error finding container 0753e8d2407b76c76797b21113ac30f2d85d9bfa9eb96ccd7a08d2847da7ebe0: Status 404 returned error can't find the container with id 0753e8d2407b76c76797b21113ac30f2d85d9bfa9eb96ccd7a08d2847da7ebe0 Nov 02 16:06:33.723394 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:33.723368 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:34.093076 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:34.093051 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p"] Nov 02 16:06:34.094865 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:06:34.094835 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71f9e57_26dd_4a32_9e6c_6cb69b8de76f.slice/crio-c4ca2204a85cdf28211a7e547efd322eef3429abeb2c386e3908b0a398fc5cd0 WatchSource:0}: Error finding container c4ca2204a85cdf28211a7e547efd322eef3429abeb2c386e3908b0a398fc5cd0: Status 404 returned error can't find the container with id c4ca2204a85cdf28211a7e547efd322eef3429abeb2c386e3908b0a398fc5cd0 Nov 02 16:06:34.471726 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:34.471688 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" event={"ID":"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7","Type":"ContainerStarted","Data":"44624221d037092cadf554514f761ea4a6bd3e2852c8897d3693738755803d4c"} Nov 02 16:06:34.471999 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:34.471745 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" event={"ID":"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7","Type":"ContainerStarted","Data":"0753e8d2407b76c76797b21113ac30f2d85d9bfa9eb96ccd7a08d2847da7ebe0"} Nov 02 16:06:34.471999 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:34.471795 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:34.473327 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:34.473303 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" event={"ID":"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f","Type":"ContainerStarted","Data":"8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160"} Nov 02 16:06:34.473450 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:34.473349 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" event={"ID":"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f","Type":"ContainerStarted","Data":"c4ca2204a85cdf28211a7e547efd322eef3429abeb2c386e3908b0a398fc5cd0"} Nov 02 16:06:34.473450 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:34.473394 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:34.582504 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:34.582452 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" podStartSLOduration=2.582437405 podStartE2EDuration="2.582437405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:06:34.58182086 +0000 UTC m=+4098.255024057" watchObservedRunningTime="2026-11-02 16:06:34.582437405 +0000 UTC m=+4098.255640600" Nov 02 16:06:34.709194 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:34.709125 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" podStartSLOduration=1.709104751 podStartE2EDuration="1.709104751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:06:34.70880336 +0000 UTC m=+4098.382006555" watchObservedRunningTime="2026-11-02 16:06:34.709104751 +0000 UTC m=+4098.382308039" Nov 02 16:06:35.490705 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:35.490672 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:35.644334 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:35.644299 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:35.797524 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:35.797438 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:37.093072 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:37.093038 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:37.245590 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:37.245543 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:39.115091 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:39.115058 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:40.530630 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:40.530590 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:41.011528 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:06:41.011493 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:06:45.067688 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:45.067650 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:45.481284 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:45.481247 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:06:45.481477 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:45.481317 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:06:49.745351 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:49.745315 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:52.599570 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:52.599482 5433 generic.go:356] "Generic (PLEG): container finished" podID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerID="8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb" exitCode=0 Nov 02 16:06:52.599570 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:52.599552 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" event={"ID":"48f374c6-0f05-4faf-9aee-99c9bb843595","Type":"ContainerDied","Data":"8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb"} Nov 02 16:06:52.600109 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:52.599920 5433 scope.go:122] "RemoveContainer" containerID="8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb" Nov 02 16:06:52.601077 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:52.601048 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:06:52.601201 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:52.601150 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:06:52.601329 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:52.601198 5433 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:06:53.606472 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:53.606438 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" event={"ID":"48f374c6-0f05-4faf-9aee-99c9bb843595","Type":"ContainerStarted","Data":"2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00"} Nov 02 16:06:53.638997 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:53.638304 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:53.638997 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:53.638435 5433 patch_prober.go:29] interesting pod/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 16:06:53.638997 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:53.638499 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 16:06:54.083833 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:54.083797 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:54.637145 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:54.637108 5433 patch_prober.go:29] interesting pod/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 16:06:54.637747 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:54.637713 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 16:06:54.770861 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:54.770818 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:55.643893 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:55.643832 5433 patch_prober.go:29] interesting pod/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 16:06:55.644609 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:55.644574 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 16:06:56.641137 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:56.641095 5433 patch_prober.go:29] interesting pod/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 16:06:56.641317 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:56.641158 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 16:06:57.635475 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:57.635430 5433 patch_prober.go:29] interesting pod/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 16:06:57.636152 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:57.636113 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 16:06:58.444696 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:58.444656 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:58.632802 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:58.632765 5433 patch_prober.go:29] interesting pod/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 16:06:58.633000 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:58.632827 5433 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 16:06:59.574236 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:59.574198 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:06:59.636886 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:59.636849 5433 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:06:59.637772 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:06:59.637756 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:07:04.218927 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:04.218888 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:08.823191 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:08.823153 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:11.011298 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:07:11.011259 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:07:13.530407 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:13.530355 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:18.224780 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:18.224746 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:22.992023 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:22.991964 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:27.668442 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:27.668363 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:32.423317 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:32.423282 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:37.140441 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:37.140399 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:39.447213 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:39.447181 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:41.014418 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:07:41.014373 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:07:41.828249 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:41.828215 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:46.572128 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:46.572084 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:51.230128 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:51.230087 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:07:55.920956 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:07:55.920915 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:00.622601 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:00.622564 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:05.335724 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:05.335691 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:10.004580 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:10.004547 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:11.013106 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:08:11.013067 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:08:14.610868 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:14.610804 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:19.452084 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:19.452033 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:24.329508 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:24.329450 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:28.866641 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:28.866602 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p"] Nov 02 16:08:28.964560 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:28.964524 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf"] Nov 02 16:08:29.224474 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:29.224438 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:29.377123 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:29.377079 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:33.866898 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:33.866859 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" podUID="b71f9e57-26dd-4a32-9e6c-6cb69b8de76f" containerName="payload-processing" containerID="cri-o://8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160" gracePeriod=25 Nov 02 16:08:33.965225 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:33.965188 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" podUID="cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7" containerName="payload-pre-processing" containerID="cri-o://44624221d037092cadf554514f761ea4a6bd3e2852c8897d3693738755803d4c" gracePeriod=25 Nov 02 16:08:34.124635 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.124607 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:08:34.125210 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.125188 5433 generic.go:356] "Generic (PLEG): container finished" podID="cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7" containerID="44624221d037092cadf554514f761ea4a6bd3e2852c8897d3693738755803d4c" exitCode=0 Nov 02 16:08:34.125313 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.125271 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" event={"ID":"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7","Type":"ContainerDied","Data":"44624221d037092cadf554514f761ea4a6bd3e2852c8897d3693738755803d4c"} Nov 02 16:08:34.126591 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.126570 5433 generic.go:356] "Generic (PLEG): container finished" podID="b71f9e57-26dd-4a32-9e6c-6cb69b8de76f" containerID="8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160" exitCode=0 Nov 02 16:08:34.126681 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.126603 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" Nov 02 16:08:34.126681 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.126649 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" event={"ID":"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f","Type":"ContainerDied","Data":"8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160"} Nov 02 16:08:34.126760 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.126696 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p" event={"ID":"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f","Type":"ContainerDied","Data":"c4ca2204a85cdf28211a7e547efd322eef3429abeb2c386e3908b0a398fc5cd0"} Nov 02 16:08:34.126760 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.126718 5433 scope.go:122] "RemoveContainer" containerID="8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160" Nov 02 16:08:34.135220 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.135197 5433 scope.go:122] "RemoveContainer" containerID="8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160" Nov 02 16:08:34.135529 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:08:34.135510 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160\": container with ID starting with 8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160 not found: ID does not exist" containerID="8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160" Nov 02 16:08:34.135606 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.135539 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160"} err="failed to get container status \"8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160\": rpc error: code = NotFound desc = could not find container \"8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160\": container with ID starting with 8278ae5add1c782c866e825f384377971066afb3b93d00aa31da83b09684f160 not found: ID does not exist" Nov 02 16:08:34.228415 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.228388 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:08:34.252624 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.252594 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-plugins-config-volume\") pod \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\" (UID: \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\") " Nov 02 16:08:34.252791 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.252642 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-kube-api-access-pgwg5\" (UniqueName: \"kubernetes.io/projected/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-kube-api-access-pgwg5\") pod \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\" (UID: \"b71f9e57-26dd-4a32-9e6c-6cb69b8de76f\") " Nov 02 16:08:34.253103 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.253073 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-plugins-config-volume" pod "b71f9e57-26dd-4a32-9e6c-6cb69b8de76f" (UID: "b71f9e57-26dd-4a32-9e6c-6cb69b8de76f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:08:34.254695 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.254675 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-kube-api-access-pgwg5" pod "b71f9e57-26dd-4a32-9e6c-6cb69b8de76f" (UID: "b71f9e57-26dd-4a32-9e6c-6cb69b8de76f"). InnerVolumeSpecName "kube-api-access-pgwg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:08:34.354261 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.354231 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-plugins-config-volume\") pod \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\" (UID: \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\") " Nov 02 16:08:34.354466 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.354283 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-kube-api-access-m59lj\" (UniqueName: \"kubernetes.io/projected/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-kube-api-access-m59lj\") pod \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\" (UID: \"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7\") " Nov 02 16:08:34.354676 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.354641 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-plugins-config-volume" pod "cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7" (UID: "cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:08:34.354676 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.354667 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:34.354676 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.354685 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgwg5\" (UniqueName: \"kubernetes.io/projected/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f-kube-api-access-pgwg5\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:34.356458 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.356435 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-kube-api-access-m59lj" pod "cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7" (UID: "cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7"). InnerVolumeSpecName "kube-api-access-m59lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:08:34.455442 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.455412 5433 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-plugins-config-volume\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:34.455442 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.455439 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m59lj\" (UniqueName: \"kubernetes.io/projected/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7-kube-api-access-m59lj\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:34.543063 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.543025 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p"] Nov 02 16:08:34.548363 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:34.548319 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-715ca8-7847d7fcc-2h82p"] Nov 02 16:08:35.132963 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:35.132875 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" Nov 02 16:08:35.132963 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:35.132876 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf" event={"ID":"cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7","Type":"ContainerDied","Data":"0753e8d2407b76c76797b21113ac30f2d85d9bfa9eb96ccd7a08d2847da7ebe0"} Nov 02 16:08:35.133501 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:35.133028 5433 scope.go:122] "RemoveContainer" containerID="44624221d037092cadf554514f761ea4a6bd3e2852c8897d3693738755803d4c" Nov 02 16:08:35.290504 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:35.290443 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf"] Nov 02 16:08:35.337148 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:35.337106 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-715ca8-7c47d6bc87-cbtgf"] Nov 02 16:08:36.454775 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:36.454743 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71f9e57-26dd-4a32-9e6c-6cb69b8de76f" path="/var/lib/kubelet/pods/b71f9e57-26dd-4a32-9e6c-6cb69b8de76f/volumes" Nov 02 16:08:36.455176 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:36.455145 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7" path="/var/lib/kubelet/pods/cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7/volumes" Nov 02 16:08:37.718308 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:37.718273 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:39.182443 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:39.182402 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:39.463872 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:39.463782 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:39.787383 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:39.787289 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:40.208688 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:40.208624 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:41.011449 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:08:41.011409 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:08:46.273476 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:46.273431 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:46.426795 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:46.426762 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:46.604476 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:46.604390 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2"] Nov 02 16:08:46.604658 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:46.604635 5433 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" containerID="cri-o://2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00" gracePeriod=30 Nov 02 16:08:46.605636 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:46.605607 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:08:51.855866 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.855839 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:08:51.927788 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.927759 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-envoy\") pod \"48f374c6-0f05-4faf-9aee-99c9bb843595\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " Nov 02 16:08:51.927963 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.927798 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-kube-api-access-jsv2c\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-kube-api-access-jsv2c\") pod \"48f374c6-0f05-4faf-9aee-99c9bb843595\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " Nov 02 16:08:51.927963 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.927818 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f374c6-0f05-4faf-9aee-99c9bb843595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f374c6-0f05-4faf-9aee-99c9bb843595-istiod-ca-cert\") pod \"48f374c6-0f05-4faf-9aee-99c9bb843595\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " Nov 02 16:08:51.927963 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.927872 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-podinfo\") pod \"48f374c6-0f05-4faf-9aee-99c9bb843595\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " Nov 02 16:08:51.927963 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.927906 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-token\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-token\") pod \"48f374c6-0f05-4faf-9aee-99c9bb843595\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " Nov 02 16:08:51.927963 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.927944 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-socket\") pod \"48f374c6-0f05-4faf-9aee-99c9bb843595\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " Nov 02 16:08:51.928306 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928003 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-certs\") pod \"48f374c6-0f05-4faf-9aee-99c9bb843595\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " Nov 02 16:08:51.928306 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928070 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-data\") pod \"48f374c6-0f05-4faf-9aee-99c9bb843595\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " Nov 02 16:08:51.928306 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928123 5433 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-credential-socket\") pod \"48f374c6-0f05-4faf-9aee-99c9bb843595\" (UID: \"48f374c6-0f05-4faf-9aee-99c9bb843595\") " Nov 02 16:08:51.928469 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928293 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-socket" pod "48f374c6-0f05-4faf-9aee-99c9bb843595" (UID: "48f374c6-0f05-4faf-9aee-99c9bb843595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:08:51.928469 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928322 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f374c6-0f05-4faf-9aee-99c9bb843595-istiod-ca-cert" pod "48f374c6-0f05-4faf-9aee-99c9bb843595" (UID: "48f374c6-0f05-4faf-9aee-99c9bb843595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:08:51.928588 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928525 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-credential-socket" pod "48f374c6-0f05-4faf-9aee-99c9bb843595" (UID: "48f374c6-0f05-4faf-9aee-99c9bb843595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:08:51.928588 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928543 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-certs" pod "48f374c6-0f05-4faf-9aee-99c9bb843595" (UID: "48f374c6-0f05-4faf-9aee-99c9bb843595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:08:51.928703 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928615 5433 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:51.928703 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928637 5433 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-credential-socket\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:51.928703 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928651 5433 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f374c6-0f05-4faf-9aee-99c9bb843595-istiod-ca-cert\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:51.928881 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.928721 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-data" pod "48f374c6-0f05-4faf-9aee-99c9bb843595" (UID: "48f374c6-0f05-4faf-9aee-99c9bb843595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:08:51.930120 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.930087 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-podinfo" pod "48f374c6-0f05-4faf-9aee-99c9bb843595" (UID: "48f374c6-0f05-4faf-9aee-99c9bb843595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 16:08:51.930120 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.930106 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-kube-api-access-jsv2c" pod "48f374c6-0f05-4faf-9aee-99c9bb843595" (UID: "48f374c6-0f05-4faf-9aee-99c9bb843595"). InnerVolumeSpecName "kube-api-access-jsv2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:08:51.930272 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.930121 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-token" pod "48f374c6-0f05-4faf-9aee-99c9bb843595" (UID: "48f374c6-0f05-4faf-9aee-99c9bb843595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:08:51.930272 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:51.930143 5433 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-envoy" pod "48f374c6-0f05-4faf-9aee-99c9bb843595" (UID: "48f374c6-0f05-4faf-9aee-99c9bb843595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:08:52.029495 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.029457 5433 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-envoy\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:52.029495 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.029492 5433 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsv2c\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-kube-api-access-jsv2c\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:52.029717 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.029508 5433 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-podinfo\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:52.029717 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.029523 5433 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-token\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:52.029717 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.029534 5433 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-workload-certs\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:52.029717 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.029545 5433 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f374c6-0f05-4faf-9aee-99c9bb843595-istio-data\") on node \"ip-10-0-135-181.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:52.237062 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.236957 5433 generic.go:356] "Generic (PLEG): container finished" podID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerID="2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00" exitCode=0 Nov 02 16:08:52.237062 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.237012 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" event={"ID":"48f374c6-0f05-4faf-9aee-99c9bb843595","Type":"ContainerDied","Data":"2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00"} Nov 02 16:08:52.237062 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.237032 5433 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" Nov 02 16:08:52.237340 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.237076 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2" event={"ID":"48f374c6-0f05-4faf-9aee-99c9bb843595","Type":"ContainerDied","Data":"ec60d83165334bfa5855d46cd51dbc632a9725d520bbf6f992098a9b9c9e749a"} Nov 02 16:08:52.237340 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.237100 5433 scope.go:122] "RemoveContainer" containerID="2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00" Nov 02 16:08:52.246730 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.246710 5433 scope.go:122] "RemoveContainer" containerID="8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb" Nov 02 16:08:52.254642 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.254624 5433 scope.go:122] "RemoveContainer" containerID="2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00" Nov 02 16:08:52.254903 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:08:52.254883 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00\": container with ID starting with 2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00 not found: ID does not exist" containerID="2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00" Nov 02 16:08:52.254950 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.254912 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00"} err="failed to get container status \"2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00\": rpc error: code = NotFound desc = could not find container \"2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00\": container with ID starting with 2e1dbd9672b2ef074f3ae8091fd86f761aeee79b40c8815d77a0c4b2426c3a00 not found: ID does not exist" Nov 02 16:08:52.254950 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.254931 5433 scope.go:122] "RemoveContainer" containerID="8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb" Nov 02 16:08:52.255189 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:08:52.255167 5433 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb\": container with ID starting with 8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb not found: ID does not exist" containerID="8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb" Nov 02 16:08:52.255249 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.255198 5433 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb"} err="failed to get container status \"8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb\": rpc error: code = NotFound desc = could not find container \"8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb\": container with ID starting with 8c023c31833909fbe53964f0284b175f4673510d6524d8263c34a7e3098ea1cb not found: ID does not exist" Nov 02 16:08:52.403189 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.403156 5433 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2"] Nov 02 16:08:52.407054 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.407026 5433 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-715ca8-openshift-default-8d9fc69b9-gzsc2"] Nov 02 16:08:52.456025 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:08:52.455994 5433 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" path="/var/lib/kubelet/pods/48f374c6-0f05-4faf-9aee-99c9bb843595/volumes" Nov 02 16:09:11.011741 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:09:11.011658 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:09:16.388161 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:09:16.388123 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:09:16.539624 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:09:16.539593 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:09:40.167952 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:09:40.167916 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:09:41.011953 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:09:41.011915 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:10:04.747002 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:04.746942 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:10:04.912670 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:04.912635 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:10:09.187558 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:09.187525 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-84664687bf-sqhrz_5680c5f2-72f0-411b-b2cd-baaae0ba3807/manager/0.log" Nov 02 16:10:10.716093 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:10.716058 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-754d6b6455-xr57v_d67daa79-d4d3-4f7e-95f0-0815963ecfc0/manager/0.log" Nov 02 16:10:11.012139 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:10:11.012055 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:10:11.492364 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:11.492334 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bs4qx_ef257c3c-af16-4b85-bfa1-3208f6b0f69e/manager/0.log" Nov 02 16:10:12.835942 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:12.835908 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57678c84d7-mnf4b_4c97324a-1055-44db-85a9-568973dec114/maas-api/0.log" Nov 02 16:10:13.788575 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:13.788544 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-bmbjm_ea0305c8-7c3e-46ca-a234-e3da552184bd/postgres/0.log" Nov 02 16:10:18.040878 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:18.040843 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5lx6c_f44d6239-c49c-4d30-839f-77bcf9eea65b/discovery/0.log" Nov 02 16:10:18.608303 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:18.608272 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848b57c455-xx8nq_15ebac82-cb6a-4535-bc22-a82bca0251d8/kube-auth-proxy/0.log" Nov 02 16:10:19.134034 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:19.133999 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4xdd8_9f7fd75c-268f-4df0-8eaf-7d9648ed4442/istio-proxy/0.log" Nov 02 16:10:19.666669 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:19.666641 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lcwhz_bac33501-52e3-44da-a6d7-4c8f3aa5a0a1/istio-proxy/0.log" Nov 02 16:10:20.197333 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:20.197306 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pws57_6b18468c-caaa-4be9-a134-7da1da1fa1c8/istio-proxy/0.log" Nov 02 16:10:20.727031 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:20.727001 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-wlmbx_2b3ce253-c2a1-4844-a2f5-2b5932a1594d/istio-proxy/0.log" Nov 02 16:10:21.691295 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:21.691261 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6tbpj_9c31e312-7bec-41e3-84b0-a6279ed4ad74/payload-pre-processing/0.log" Nov 02 16:10:21.909256 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:21.909227 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z9f75_e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f/payload-processing/0.log" Nov 02 16:10:23.033212 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:23.033181 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88_9590d5ee-9ed8-4f51-bd39-b044c4a7f80f/main/0.log" Nov 02 16:10:23.302918 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:23.302838 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-f9l8p_e185e809-cc45-46d1-9912-e0bfc97ebb22/main/0.log" Nov 02 16:10:23.517296 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:23.517266 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-85zn5_3ae29092-e319-49bd-ae57-741c217a1e03/main/0.log" Nov 02 16:10:23.734653 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:23.734620 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-4p5hr_65866332-3ec1-4250-8f95-67bcf624745d/main/0.log" Nov 02 16:10:24.420171 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:24.420138 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5_89a646d3-ed9b-4cff-9d54-1ba94e3bfc00/main/0.log" Nov 02 16:10:34.682176 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:34.682138 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-84664687bf-sqhrz_5680c5f2-72f0-411b-b2cd-baaae0ba3807/manager/0.log" Nov 02 16:10:35.886919 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:35.886885 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-754d6b6455-xr57v_d67daa79-d4d3-4f7e-95f0-0815963ecfc0/manager/0.log" Nov 02 16:10:36.522393 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:36.522366 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-bs4qx_ef257c3c-af16-4b85-bfa1-3208f6b0f69e/manager/0.log" Nov 02 16:10:37.608219 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:37.608172 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57678c84d7-mnf4b_4c97324a-1055-44db-85a9-568973dec114/maas-api/0.log" Nov 02 16:10:38.570257 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:38.570224 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-bmbjm_ea0305c8-7c3e-46ca-a234-e3da552184bd/postgres/0.log" Nov 02 16:10:40.432094 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:40.432053 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:10:41.016542 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:10:41.016499 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:10:42.180378 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:42.180344 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5lx6c_f44d6239-c49c-4d30-839f-77bcf9eea65b/discovery/0.log" Nov 02 16:10:42.728775 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:42.728746 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848b57c455-xx8nq_15ebac82-cb6a-4535-bc22-a82bca0251d8/kube-auth-proxy/0.log" Nov 02 16:10:43.192656 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:43.192628 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4xdd8_9f7fd75c-268f-4df0-8eaf-7d9648ed4442/istio-proxy/0.log" Nov 02 16:10:43.686206 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:43.686175 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lcwhz_bac33501-52e3-44da-a6d7-4c8f3aa5a0a1/istio-proxy/0.log" Nov 02 16:10:44.202191 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:44.202158 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pws57_6b18468c-caaa-4be9-a134-7da1da1fa1c8/istio-proxy/0.log" Nov 02 16:10:44.725516 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:44.725476 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-wlmbx_2b3ce253-c2a1-4844-a2f5-2b5932a1594d/istio-proxy/0.log" Nov 02 16:10:45.601586 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:45.601550 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6tbpj_9c31e312-7bec-41e3-84b0-a6279ed4ad74/payload-pre-processing/0.log" Nov 02 16:10:45.829031 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:45.829002 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z9f75_e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f/payload-processing/0.log" Nov 02 16:10:46.668191 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:46.668161 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7rq88_9590d5ee-9ed8-4f51-bd39-b044c4a7f80f/main/0.log" Nov 02 16:10:46.944889 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:46.944808 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-f9l8p_e185e809-cc45-46d1-9912-e0bfc97ebb22/main/0.log" Nov 02 16:10:47.191625 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:47.191593 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-85zn5_3ae29092-e319-49bd-ae57-741c217a1e03/main/0.log" Nov 02 16:10:47.407313 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:47.407268 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-4p5hr_65866332-3ec1-4250-8f95-67bcf624745d/main/0.log" Nov 02 16:10:48.119344 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:10:48.119311 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-6pnw5_89a646d3-ed9b-4cff-9d54-1ba94e3bfc00/main/0.log" Nov 02 16:11:11.011302 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:11:11.011261 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:11:16.049443 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:16.049410 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-754d6b6455-xr57v_d67daa79-d4d3-4f7e-95f0-0815963ecfc0/manager/0.log" Nov 02 16:11:16.544693 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:16.544663 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-754d6b6455-xr57v_d67daa79-d4d3-4f7e-95f0-0815963ecfc0/manager/0.log" Nov 02 16:11:17.292780 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:17.292751 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z9f75_e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f/payload-processing/0.log" Nov 02 16:11:17.519867 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:17.519840 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z9f75_e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f/payload-processing/0.log" Nov 02 16:11:35.530601 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:35.530565 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:11:38.604309 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:38.604264 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:11:40.775816 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:40.775772 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:11:41.011961 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:11:41.011924 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:11:52.966501 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:52.966468 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-754d6b6455-xr57v_d67daa79-d4d3-4f7e-95f0-0815963ecfc0/manager/0.log" Nov 02 16:11:53.516117 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:53.516083 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-754d6b6455-xr57v_d67daa79-d4d3-4f7e-95f0-0815963ecfc0/manager/0.log" Nov 02 16:11:54.272142 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:54.272109 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z9f75_e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f/payload-processing/0.log" Nov 02 16:11:54.481892 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:54.481860 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z9f75_e276f1d0-b6e5-4dba-8883-a5bfc3a6c77f/payload-processing/0.log" Nov 02 16:11:54.806954 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:54.806918 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:11:54.954417 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:11:54.954383 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:12:03.374508 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:03.374428 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-69x5r_ed84914e-49d4-4f14-877b-ca1b0cdff144/global-pull-secret-syncer/0.log" Nov 02 16:12:06.359380 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:06.359338 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" Nov 02 16:12:08.104368 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:08.104328 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-181.us-west-2.compute.internal_010a8efd81cfe8476c37b8d73ee870b2/haproxy/0.log" Nov 02 16:12:11.011006 ip-10-0-135-181 kubenswrapper[5433]: E1102 16:12:11.010955 5433 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4njs7" containerName="extractor" Nov 02 16:12:15.001032 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.000992 5433 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62"] Nov 02 16:12:15.001487 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001472 5433 cpu_manager.go:414] "containerMap: removing container" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" Nov 02 16:12:15.001532 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001490 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" Nov 02 16:12:15.001532 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001506 5433 cpu_manager.go:414] "containerMap: removing container" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" Nov 02 16:12:15.001532 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001511 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" Nov 02 16:12:15.001532 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001522 5433 cpu_manager.go:414] "containerMap: removing container" podUID="b71f9e57-26dd-4a32-9e6c-6cb69b8de76f" containerName="payload-processing" Nov 02 16:12:15.001532 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001527 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71f9e57-26dd-4a32-9e6c-6cb69b8de76f" containerName="payload-processing" Nov 02 16:12:15.001721 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001537 5433 cpu_manager.go:414] "containerMap: removing container" podUID="cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7" containerName="payload-pre-processing" Nov 02 16:12:15.001721 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001545 5433 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7" containerName="payload-pre-processing" Nov 02 16:12:15.001721 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001658 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" Nov 02 16:12:15.001721 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001665 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f374c6-0f05-4faf-9aee-99c9bb843595" containerName="istio-proxy" Nov 02 16:12:15.001721 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001672 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71f9e57-26dd-4a32-9e6c-6cb69b8de76f" containerName="payload-processing" Nov 02 16:12:15.001721 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.001678 5433 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4d7283-5eba-40e7-9c1e-e0a7b2d677c7" containerName="payload-pre-processing" Nov 02 16:12:15.004720 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.004700 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.054370 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.054333 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2cdf\"/\"kube-root-ca.crt\"" Nov 02 16:12:15.097903 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.097872 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-proc\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.098082 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.097917 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hmt\" (UniqueName: \"kubernetes.io/projected/1135d5be-b273-4912-8349-f1331c72512f-kube-api-access-b9hmt\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.098082 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.098020 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-sys\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.098082 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.098047 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-podres\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.098197 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.098152 5433 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-lib-modules\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.100832 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.100805 5433 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2cdf\"/\"openshift-service-ca.crt\"" Nov 02 16:12:15.101060 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.100892 5433 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f2cdf\"/\"default-dockercfg-bcp6z\"" Nov 02 16:12:15.103147 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.103125 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62"] Nov 02 16:12:15.199469 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.199425 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-proc\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.199663 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.199487 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hmt\" (UniqueName: \"kubernetes.io/projected/1135d5be-b273-4912-8349-f1331c72512f-kube-api-access-b9hmt\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.199663 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.199546 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-sys\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.199663 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.199574 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-proc\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.199663 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.199610 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-sys\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.199663 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.199636 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-podres\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.199886 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.199681 5433 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-lib-modules\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.199886 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.199758 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-podres\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.199886 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.199764 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1135d5be-b273-4912-8349-f1331c72512f-lib-modules\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.257452 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.257374 5433 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hmt\" (UniqueName: \"kubernetes.io/projected/1135d5be-b273-4912-8349-f1331c72512f-kube-api-access-b9hmt\") pod \"perf-node-gather-daemonset-rsd62\" (UID: \"1135d5be-b273-4912-8349-f1331c72512f\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.317845 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.317812 5433 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:15.684232 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:15.684206 5433 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62"] Nov 02 16:12:15.687779 ip-10-0-135-181 kubenswrapper[5433]: W1102 16:12:15.687750 5433 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1135d5be_b273_4912_8349_f1331c72512f.slice/crio-22feae49928d192b1cfd91daf94de9c37ac161b1ed1c0ee30401646b722ca85c WatchSource:0}: Error finding container 22feae49928d192b1cfd91daf94de9c37ac161b1ed1c0ee30401646b722ca85c: Status 404 returned error can't find the container with id 22feae49928d192b1cfd91daf94de9c37ac161b1ed1c0ee30401646b722ca85c Nov 02 16:12:16.287999 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:16.287939 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" event={"ID":"1135d5be-b273-4912-8349-f1331c72512f","Type":"ContainerStarted","Data":"2a0c440a9edb110e99badf387f5238ddaddf46b9701fbd0bc81896c3f85161a9"} Nov 02 16:12:16.287999 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:16.288004 5433 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" event={"ID":"1135d5be-b273-4912-8349-f1331c72512f","Type":"ContainerStarted","Data":"22feae49928d192b1cfd91daf94de9c37ac161b1ed1c0ee30401646b722ca85c"} Nov 02 16:12:16.288547 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:16.288104 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:16.401103 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:16.401047 5433 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" podStartSLOduration=2.401033294 podStartE2EDuration="2.401033294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 16:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:12:16.399914234 +0000 UTC m=+4440.073117429" watchObservedRunningTime="2026-11-02 16:12:16.401033294 +0000 UTC m=+4440.074236488" Nov 02 16:12:21.539044 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:21.539016 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jqdhk_bd0ae468-b91d-4121-baca-0cc3836997ae/kube-state-metrics/0.log" Nov 02 16:12:21.933492 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:21.933459 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jqdhk_bd0ae468-b91d-4121-baca-0cc3836997ae/kube-rbac-proxy-main/0.log" Nov 02 16:12:22.303314 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:22.303240 5433 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-rsd62" Nov 02 16:12:22.327124 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:22.327096 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jqdhk_bd0ae468-b91d-4121-baca-0cc3836997ae/kube-rbac-proxy-self/0.log" Nov 02 16:12:22.726528 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:22.726503 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-74bdbddd8-wl9lp_e30a530f-5e3c-48f7-8872-f0b58d08a91e/metrics-server/0.log" Nov 02 16:12:23.120017 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:23.119917 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-mxp9b_5a3d2fcf-47c8-4c96-b493-4551ca3a9bf6/monitoring-plugin/0.log" Nov 02 16:12:23.514442 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:23.514413 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d4l7z_ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/node-exporter/0.log" Nov 02 16:12:23.902867 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:23.902841 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d4l7z_ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/kube-rbac-proxy/0.log" Nov 02 16:12:24.292310 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:24.292224 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d4l7z_ea7b659d-dcaa-40e1-ac29-5cdc7a619a4f/init-textfile/0.log" Nov 02 16:12:27.059673 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:27.059638 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-nhf9q_e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/kube-rbac-proxy-main/0.log" Nov 02 16:12:27.458012 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:27.457960 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-nhf9q_e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/kube-rbac-proxy-self/0.log" Nov 02 16:12:27.857308 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:27.857214 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-nhf9q_e4b3ae42-1fb7-4068-80bd-a8c94e1d35f3/openshift-state-metrics/0.log" Nov 02 16:12:32.268385 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:32.268349 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:12:32.418832 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:32.418797 5433 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4522z_2616b3d7-11af-4008-99e7-44a5c575e85a/konnectivity-agent/0.log" limit=1024 Nov 02 16:12:33.547643 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:33.547610 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b4c77bc7-qtd5b_83a2eda6-8157-428a-b0f3-c2cc77563252/thanos-query/0.log" Nov 02 16:12:33.943341 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:33.943311 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b4c77bc7-qtd5b_83a2eda6-8157-428a-b0f3-c2cc77563252/kube-rbac-proxy-web/0.log" Nov 02 16:12:34.339182 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:34.339096 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b4c77bc7-qtd5b_83a2eda6-8157-428a-b0f3-c2cc77563252/kube-rbac-proxy/0.log" Nov 02 16:12:34.729635 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:34.729605 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b4c77bc7-qtd5b_83a2eda6-8157-428a-b0f3-c2cc77563252/prom-label-proxy/0.log" Nov 02 16:12:35.120256 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:35.120188 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b4c77bc7-qtd5b_83a2eda6-8157-428a-b0f3-c2cc77563252/kube-rbac-proxy-rules/0.log" Nov 02 16:12:35.510922 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:35.510877 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b4c77bc7-qtd5b_83a2eda6-8157-428a-b0f3-c2cc77563252/kube-rbac-proxy-metrics/0.log" Nov 02 16:12:37.988405 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:37.988376 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5lx6c_f44d6239-c49c-4d30-839f-77bcf9eea65b/discovery/0.log" Nov 02 16:12:38.494606 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:38.494575 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848b57c455-xx8nq_15ebac82-cb6a-4535-bc22-a82bca0251d8/kube-auth-proxy/0.log" Nov 02 16:12:38.901051 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:38.901015 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4xdd8_9f7fd75c-268f-4df0-8eaf-7d9648ed4442/istio-proxy/0.log" Nov 02 16:12:39.308063 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:39.307956 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lcwhz_bac33501-52e3-44da-a6d7-4c8f3aa5a0a1/istio-proxy/0.log" Nov 02 16:12:39.725690 ip-10-0-135-181 kubenswrapper[5433]: I1102 16:12:39.725661 5433 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pws57_6b18468c-caaa-4be9-a134-7da1da1fa1c8/istio-proxy/0.log"